Sep 16 12:46:15.775918 ip-10-0-137-128 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:46:15.776070 ip-10-0-137-128 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:46:15.776165 ip-10-0-137-128 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:46:15.777267 ip-10-0-137-128 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:46:25.964822 ip-10-0-137-128 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 12:46:25.964838 ip-10-0-137-128 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 42259d04654d47d78647b8a3ddf7a4dd --
Sep 16 12:48:13.839901 ip-10-0-137-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:48:14.220810 ip-10-0-137-128 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:48:14.220810 ip-10-0-137-128 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:48:14.220810 ip-10-0-137-128 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:48:14.220810 ip-10-0-137-128 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:48:14.220810 ip-10-0-137-128 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:48:14.224019 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.223927    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230882    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230903    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230908    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230911    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230915    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230918    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.230911 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230921    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230926    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230931    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230934    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230937    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230940    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230943    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230945    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230948    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230950    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230953    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230956    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230958    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230961    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230964    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230966    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230969    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230972    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230974    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230977    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.231176 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230980    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230983    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230986    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230988    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230991    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.230998    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231000    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231003    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231006    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231008    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231010    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231013    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231015    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231018    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231021    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231024    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231027    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231030    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231032    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.231673 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231035    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231037    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231040    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231043    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231046    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231048    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231051    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231053    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231056    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231059    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231062    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231065    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231067    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231070    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231072    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231075    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231077    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231080    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231083    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231085    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.232155 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231088    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231090    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231093    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231096    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231098    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231101    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231103    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231106    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231109    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231112    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231114    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231117    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231120    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231122    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231125    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231127    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231130    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231132    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231136    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231138    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.232652 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231141    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231143    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231146    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231149    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231151    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231154    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231157    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231161    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231165    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231168    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231171    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231174    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231178    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.231182    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232712    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232721    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232724    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232727    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232730    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.233158 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232733    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232737    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232741    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232744    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232747    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232751    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232754    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232757    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232760    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232763    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232766    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232768    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232772    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232775    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232777    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232780    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232785    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232788    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232791    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.233627 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232794    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232797    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232799    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232802    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232806    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232809    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232812    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232814    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232817    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232819    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232822    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232825    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232827    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232830    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232832    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232835    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232837    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232840    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232843    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.234130 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232845    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232847    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232850    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232852    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232855    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232857    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232860    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232864    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232867    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232870    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232873    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232876    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232878    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232881    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232883    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232886    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232888    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232891    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232894    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232896    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.234609 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232899    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232901    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232904    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232906    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232909    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232912    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232914    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232917    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232919    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232922    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232924    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232927    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232930    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232932    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232935    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232937    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232941    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232943    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232946    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232948    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.235195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232951    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232955    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232958    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232960    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232963    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232965    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232968    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232970    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232974    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232976    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232979    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232981    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232983    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232986    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232988    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.232991    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233374    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233379    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233381    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233384    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.235724 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233387    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233389    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233392    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233395    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233397    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233400    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233402    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233405    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233408    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233410    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233412    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233415    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233423    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233426    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233429    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233431    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233434    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233437    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233439    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233442    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.236212 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233445    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233447    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233450    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233453    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233455    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233458    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233460    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233463    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233465    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233467    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233470    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233472    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233475    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233478    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233480    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233483    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233485    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233488    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233491    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233493    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.236711 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233496    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233498    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233501    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233504    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233506    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233510    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233512    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233517    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233520    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233523    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233527    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233530    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233533    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233536    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233538    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233540    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233543    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233546    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233548    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.237200 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233550    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233553    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233555    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233572    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233575    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233579    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233582    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233584    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233587    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233590    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233595    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233600    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233602    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233605    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233608    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233610    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233613    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233616    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233618    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233621    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.237704 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233624    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233626    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233629    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233632    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233634    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233638    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233642    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233646    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233649    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233652    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233655    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233658    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233660    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233663    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233665    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.233668    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233746    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233785    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233795    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233799    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:48:14.238194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233803    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233806    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233811    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233817    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233821    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233824    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233827    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233830    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233834    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233836    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233839    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233842    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233846    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233849    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233853    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233856    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233859    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233862    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233865    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233869    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233872    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233875    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233878    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233881    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:48:14.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233885    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233888    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233891    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233894    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233898    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233901    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233904    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233907    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233910    2579 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233913    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233918    2579 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233923    2579 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233926    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233929    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233932    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233936    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233939    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233942    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233945    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233948    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233951    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233954    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233958    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233961    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233963    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:48:14.239304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233966    2579 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233970    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233973    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233976    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233979    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233982    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233985    2579 flags.go:64] FLAG: --help="false"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233988    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233991    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233994    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.233997    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234000    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234003    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234006    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234009    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234011    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234014    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234017    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234022    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234025    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234028    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234030    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234034    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234036    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:48:14.239975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234039    2579 flags.go:64] FLAG: --lock-file=""
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234042    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234045    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234048    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234053    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234056    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234059    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234062    2579 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234065    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234068    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234071    2579 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234074    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234078    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234081    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234085    2579 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234088    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234091    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234093    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234096    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234099    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234102    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234105    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234113    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234116    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234119    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:48:14.240573 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234122    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234126    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234131    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234134    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234138    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234141    2579 flags.go:64] FLAG: --port="10250"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234144    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234147    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091f468321dc95c83"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234150    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234153    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234156    2579 flags.go:64] FLAG: --register-node="true"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234159    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234163    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234166    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234169    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234172    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234175    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234178    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234181    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234184    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234187    2579 flags.go:64] FLAG: --runonce="false"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234189    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234193    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234196    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234198    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:48:14.241247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234201    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234204    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234207    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234210    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234213    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234216    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234219    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234222    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234227    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234230    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234232    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234238    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234240    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234243    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234247    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234250    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234253    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234256    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234259    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234262    2579 flags.go:64] FLAG: --v="2"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234266    2579 flags.go:64] FLAG: --version="false"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234270    2579 flags.go:64] FLAG: --vmodule=""
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234274    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.234277    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234404    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.241882 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234407    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234410    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234413    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234416    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234418    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234421    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234423    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234426    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234429    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234432    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234434    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234437    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234440    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234443    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234446    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234452    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234455    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234457    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234460    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234463    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.242517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234465    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234468    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234470    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234473    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234476    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234478    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234481    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234483    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234486    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234490    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234492    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234495    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234498    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234501    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234504    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234508    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234512    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234515    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234517    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234520    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.243020 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234523    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234525    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234528    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234531    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234533    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234536    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234538    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234543    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234546    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234549    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234552    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234554    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234569    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234572    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234575    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234578    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234580    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234583    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234585    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.243517 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234588    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234592    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234596    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234601    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234604    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234607    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234610    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234613    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234617    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234620    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234622    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234625    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234628    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234631    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234634    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234636    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234639    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234641    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234644    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234647    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.244019 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234652    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234654    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234657    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234660    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234662    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234665    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234667    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234670    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234674    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234677    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234681    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234684    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234687    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234691    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234694    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234697    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234701    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234704    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.244518 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.234707    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.235244    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.242590    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.242606    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.243323    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.243330    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243387    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243393    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243396    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243399    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243402    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243405    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243407    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243410    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243412    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.245083 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243415    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243418    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243421    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243423    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243426    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243429    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243431    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243434    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243436    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243439    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243442    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243445    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243447    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243450    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243452    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243456    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243458    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243461    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243464    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243466    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.245464 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243470    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243473    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243476    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243478    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243481    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243483    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243486    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243489    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243491    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243494    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243496    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243499    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243501    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243504    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243506    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243509    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243511    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243514    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243517    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.245964 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243519    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243522    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243525    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243527    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243530    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243533    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243536    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243539    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243541    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243546    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243550    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243553    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243556    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243578    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243581    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243584    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243586    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243589    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243592    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.246421 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243595    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243597    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243600    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243602    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243605    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243607    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243610    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243612    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243615    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243618    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243620    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243623    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243625    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243628    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243630    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243633    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243635    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243638    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243641    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243644    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.246900 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243647    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243649    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243652    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243656    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243658    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243661    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243665    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243668    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243671    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243674    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243678    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243682    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.243687    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243792    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:48:14.247379 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243797    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243800    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243802    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243805    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243807    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243810    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243813    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243816    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243818    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243821    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243823    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243826    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243830    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243834    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243837    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243840    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243843    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243846    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243849    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:48:14.247744 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243852    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243855    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243858    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243861    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243864    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243867    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243869    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243872    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243875    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243877    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243880    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243882    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243885    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243888    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243890    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243893    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243895    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243898    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243900    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:48:14.248214 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243903    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243905    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243908    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243910    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243913    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243915    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243918    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243921    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243923    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243926    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243928    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243931    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243933    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243936    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243938    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243941    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243944    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243947    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243949    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243953    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:48:14.248745 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243955    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243958    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243961    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243963    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243966    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243968    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243971    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243973    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243976    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243978    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243981    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243983    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243986    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243989    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243991    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243994    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243996    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.243999    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244001    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244005    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:48:14.249226 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244015    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244019    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244022    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244025    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244029    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244033    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244037    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244039    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244042    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244045    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244049    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244052    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244054    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244057    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244059    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244062    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244065    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244067    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244070    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:48:14.249740 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:14.244072    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.244077    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.244193    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.246939    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.248615    2579 server.go:1019] "Starting client certificate rotation"
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.248725    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:48:14.250208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.248765    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:48:14.250894 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.250880    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:48:14.257599 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.257574    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:48:14.269840 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.269819    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:48:14.272977 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.272960    2579 log.go:25] "Validated CRI v1 image API"
Sep 16 12:48:14.274430 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.274416    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:48:14.275788 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.275769    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:48:14.278380 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.278357    2579 fs.go:135] Filesystem UUIDs: map[574bc4b6-c669-4b67-b287-dcf2ddd10213:/dev/nvme0n1p3 687589f1-4eed-4285-9c6f-551b3bf94115:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 12:48:14.278425 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.278380    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:48:14.283400 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.283285    2579 manager.go:216] Machine: {Timestamp:2026-09-16 12:48:14.281991658 +0000 UTC m=+0.320790802 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099133 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec296dc46950c01916ff324a3d106a84 SystemUUID:ec296dc4-6950-c019-16ff-324a3d106a84 BootID:42259d04-654d-47d7-8647-b8a3ddf7a4dd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:58:91:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:58:91:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:a3:8d:86:cc:d8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:48:14.283400 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.283396    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:48:14.283509 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.283483    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:48:14.286478 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.286453    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:48:14.286634 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.286482    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:48:14.287796 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.287785    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:48:14.287831 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.287798    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:48:14.287831 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.287807    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:48:14.288508 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.288498    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:48:14.289323 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.289312    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:48:14.293089 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293076    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:48:14.293217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293208    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:48:14.293790 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293755    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:48:14.293790 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293777    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:48:14.293864 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293793    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:48:14.293864 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293804    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:48:14.293864 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.293813    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:48:14.294303 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.294290    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:48:14.294350 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.294308    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:48:14.303174 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.302950    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:48:14.305320 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.305302    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:48:14.305409 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.305332    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:48:14.305663 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.305644    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:48:14.305706 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.305668    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:48:14.305830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.305816    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4jnj"
Sep 16 12:48:14.306970 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306954    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306972    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306979    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306984    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306993    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.306998    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307011    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307016    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:48:14.307023 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307023    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:48:14.307226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307029    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:48:14.307226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307038    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:48:14.307226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307043    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:48:14.307701 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307692    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:48:14.307732 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.307702    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:48:14.311304 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.311291    2579 server.go:1268] "Started kubelet"
Sep 16 12:48:14.311401 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.311372    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:48:14.311454 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.311376    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:48:14.311454 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.311445    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:48:14.312288 ip-10-0-137-128 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:48:14.313089 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.313072    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:48:14.313905 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.313766    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:48:14.314491 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.314477    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4jnj"
Sep 16 12:48:14.315190 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.315175    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:48:14.318439 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.318424    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:48:14.319084 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.319066    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:48:14.320071 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.319235    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.320071 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320027    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:48:14.320219 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320210    2579 factory.go:55] Registering systemd factory
Sep 16 12:48:14.320254 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320222    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:48:14.320292 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320253    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:48:14.320292 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320277    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:48:14.320292 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320288    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:48:14.320441 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320429    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:48:14.320485 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320439    2579 factory.go:153] Registering CRI-O factory
Sep 16 12:48:14.320485 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320441    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:48:14.320485 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320460    2579 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:48:14.320587 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320510    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:48:14.320587 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320528    2579 factory.go:103] Registering Raw factory
Sep 16 12:48:14.320649 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.320589    2579 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:48:14.321005 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.319938    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-128.ec2.internal.18d5cdf88aa48a38  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-128.ec2.internal,UID:ip-10-0-137-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-128.ec2.internal,},FirstTimestamp:2026-09-16 12:48:14.311262776 +0000 UTC m=+0.350061891,LastTimestamp:2026-09-16 12:48:14.311262776 +0000 UTC m=+0.350061891,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-128.ec2.internal,}"
Sep 16 12:48:14.321126 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.321007    2579 manager.go:349] Starting recovery of all containers
Sep 16 12:48:14.326585 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.326486    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:14.330441 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.330409    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-128.ec2.internal\" not found" node="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.332382 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.332359    2579 manager.go:354] Recovery completed
Sep 16 12:48:14.337751 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.337634    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.339425 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.339408    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.339487 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.339439    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.339487 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.339450    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.340072 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340055    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:48:14.340072 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340069    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:48:14.340201 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340080    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.340201 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340089    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:48:14.340798 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340781    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.340875 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340814    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.340875 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.340825    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.342175 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.342160    2579 policy_none.go:49] "None policy: Start"
Sep 16 12:48:14.342175 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.342177    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:48:14.342284 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.342187    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:48:14.344156 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.344144    2579 policy_none.go:47] "Start"
Sep 16 12:48:14.372335 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372318    2579 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.372359    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372372    2579 server.go:87] "Starting device plugin registration server"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372760    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372778    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372864    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372945    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.372954    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.373638    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:48:14.394411 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.373699    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.450431 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.450392    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:48:14.451708 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.451689    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:48:14.451849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.451743    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:48:14.451849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.451777    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:48:14.451955 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.451883    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:48:14.454371 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.454353    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:14.473665 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.473550    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.474456 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.474439    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.474526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.474474    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.474526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.474491    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.474526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.474516    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.480654 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.480636    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.480716 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.480662    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-128.ec2.internal\": node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.496994 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.496969    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.552798 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.552753    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"]
Sep 16 12:48:14.552898 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.552858    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.553871 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.553854    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.553951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.553884    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.553951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.553896    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.555945 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.555933    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.556060 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.556110 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556073    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.556695 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556680    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.556695 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556693    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.556803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556706    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.556803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556711    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.556803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556716    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.556803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.556719    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.558752 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.558738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.558812 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.558763    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:48:14.559437 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.559423    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:48:14.559501 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.559450    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:48:14.559501 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.559462    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:48:14.579173 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.579150    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-128.ec2.internal\" not found" node="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.582484 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.582468    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-128.ec2.internal\" not found" node="ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.597059 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.597037    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.622380 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.622358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bfe345cfe49aded3ae47813e63145e8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-128.ec2.internal\" (UID: \"3bfe345cfe49aded3ae47813e63145e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.622452 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.622384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.622452 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.622404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.697499 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.697476    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.722636 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.722706 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.722706 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.722771 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bfe345cfe49aded3ae47813e63145e8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-128.ec2.internal\" (UID: \"3bfe345cfe49aded3ae47813e63145e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.722803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bfe345cfe49aded3ae47813e63145e8-config\") pod \"kube-apiserver-proxy-ip-10-0-137-128.ec2.internal\" (UID: \"3bfe345cfe49aded3ae47813e63145e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.722803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.722791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e982e11182de362b0fd46b40e96f40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal\" (UID: \"7e982e11182de362b0fd46b40e96f40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.798104 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.798012    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:14.881605 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.881550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.884705 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:14.884688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:14.899411 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:14.899066    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:15.003710 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.003661    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:15.110275 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.110211    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-128.ec2.internal\" not found"
Sep 16 12:48:15.169037 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.169012    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:15.220221 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.220196    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:15.235413 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.235394    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:48:15.237300 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.237285    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal"
Sep 16 12:48:15.245364 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.245343    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:48:15.247675 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.247657    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:48:15.247808 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.247790    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:48:15.247849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.247823    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:48:15.247849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.247824    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:48:15.297432 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.297393    2579 apiserver.go:52] "Watching apiserver"
Sep 16 12:48:15.319184 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.319156    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:48:15.320388 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.320353    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:43:14 +0000 UTC" deadline="2028-06-26 18:58:53.280958916 +0000 UTC"
Sep 16 12:48:15.320388 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.320381    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15582h10m37.960580598s"
Sep 16 12:48:15.320776 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.320757    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:48:15.321094 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.321075    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z","openshift-cluster-node-tuning-operator/tuned-gg5w4","openshift-dns/node-resolver-c9wd8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal","openshift-multus/multus-mmr78","openshift-multus/network-metrics-daemon-57zxt","openshift-network-diagnostics/network-check-target-p2tq4","kube-system/global-pull-secret-syncer-74lv5","kube-system/konnectivity-agent-rc64q","kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal","openshift-image-registry/node-ca-8t554","openshift-multus/multus-additional-cni-plugins-9qvkp","openshift-network-operator/iptables-alerter-fkq62","openshift-ovn-kubernetes/ovnkube-node-g7j85"]
Sep 16 12:48:15.323522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.323504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.326419 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.326398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.329166 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.328521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.331022 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.331002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.333234 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.333535 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333513    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.333657 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.333657 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333563    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.333657 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333614    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:48:15.333818 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.333671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.334693 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.334782 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334766    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.334835 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:48:15.334877 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rr7p\""
Sep 16 12:48:15.334877 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334806    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kgdtv\""
Sep 16 12:48:15.334956 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.334899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjt8l\""
Sep 16 12:48:15.335040 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.335013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ns8b\""
Sep 16 12:48:15.335239 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.335224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.335479 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.335467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.335554 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.335535    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:15.336718 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.336704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.337187 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.337173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:48:15.337559 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.337546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.337922 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.337904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:15.338025 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.337987    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:15.338673 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.338657    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.338786 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.338769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vh2jm\""
Sep 16 12:48:15.339875 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.339857    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:48:15.339952 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.339913    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:48:15.340149 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.340133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.340217 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.340193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:15.342409 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.342386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.344975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.344956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.346627 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.346606    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:48:15.346738 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.346722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qtnw4\""
Sep 16 12:48:15.346980 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.346965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:48:15.347526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.347507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.347677 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.347658    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:48:15.347770 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.347666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.348283 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.348254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:48:15.348891 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.348874    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.349131 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.349094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6ctx\""
Sep 16 12:48:15.349408 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.349388    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:48:15.349601 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.349555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:48:15.350108 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.350090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.351045 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.351029    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:48:15.351515 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.351502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:48:15.351951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.351938    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:48:15.352100 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.352088    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zqxjv\""
Sep 16 12:48:15.354788 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.354762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:48:15.354788 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.354774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2lxbf\""
Sep 16 12:48:15.354911 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.354863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:48:15.372017 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.371994    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmzx8"
Sep 16 12:48:15.386343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.386316    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vmzx8"
Sep 16 12:48:15.421394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.421372    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:48:15.426105 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgg2\" (UniqueName: \"kubernetes.io/projected/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-kube-api-access-4hgg2\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.426228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c704c5a-edcd-423d-831d-2cef17a01ecd-host\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.426228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-multus\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.426228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovn-node-metrics-cert\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.426228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.426228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhq8\" (UniqueName: \"kubernetes.io/projected/d218c9f2-9a63-4429-84b7-67d6d88169b9-kube-api-access-sdhq8\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhm6j\" (UniqueName: \"kubernetes.io/projected/9ce33184-76d1-4fdc-a7b2-d979b3f81245-kube-api-access-dhm6j\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-sys\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-etc-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bqc\" (UniqueName: \"kubernetes.io/projected/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-kube-api-access-l5bqc\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.426421 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.426619 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-sys-fs\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.426619 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-netns\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.426619 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-system-cni-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.426619 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5h8\" (UniqueName: \"kubernetes.io/projected/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-kube-api-access-dx5h8\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.426619 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-log-socket\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-bin\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-kubernetes\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7sd\" (UniqueName: \"kubernetes.io/projected/9c704c5a-edcd-423d-831d-2cef17a01ecd-kube-api-access-rj7sd\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-systemd\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-tuned\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-netns\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-ovn\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d218c9f2-9a63-4429-84b7-67d6d88169b9-tmp-dir\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c704c5a-edcd-423d-831d-2cef17a01ecd-serviceca\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-run\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-kubelet\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-netd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjss\" (UniqueName: \"kubernetes.io/projected/2b5a509a-af25-4b9b-a670-d8b9473b90d4-kube-api-access-xqjss\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.426988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-systemd-units\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-node-log\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427066 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-device-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-tmp\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-host\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-bin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-conf-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-env-overrides\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d218c9f2-9a63-4429-84b7-67d6d88169b9-hosts-file\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ce33184-76d1-4fdc-a7b2-d979b3f81245-iptables-alerter-script\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-var-lib-kubelet\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-k8s-cni-cncf-io\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-dbus\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.427394 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b6e0fb6-82c9-4ed3-9734-16bb78615874-agent-certs\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-conf\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cnibin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-slash\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-systemd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cnibin\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysconfig\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cni-binary-copy\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-socket-dir-parent\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-kubelet\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-hostroot\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-var-lib-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-system-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ce33184-76d1-4fdc-a7b2-d979b3f81245-host-slash\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.428130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-etc-kubernetes\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-config\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-script-lib\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-registration-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtwb\" (UniqueName: \"kubernetes.io/projected/4474ff46-c83c-4129-aee8-4109818d6fd7-kube-api-access-qqtwb\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.427987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-modprobe-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-lib-modules\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-daemon-config\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-kubelet-config\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e0fb6-82c9-4ed3-9734-16bb78615874-konnectivity-ca\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-socket-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-multus-certs\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.428830 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.428258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkk5n\" (UniqueName: \"kubernetes.io/projected/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-kube-api-access-rkk5n\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.529381 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-systemd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529381 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cnibin\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysconfig\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cni-binary-copy\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-socket-dir-parent\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-systemd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cnibin\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-kubelet\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysconfig\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-kubelet\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-hostroot\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-var-lib-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-socket-dir-parent\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-hostroot\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-var-lib-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.529679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-system-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ce33184-76d1-4fdc-a7b2-d979b3f81245-host-slash\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-etc-kubernetes\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-system-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-config\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-etc-kubernetes\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-script-lib\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ce33184-76d1-4fdc-a7b2-d979b3f81245-host-slash\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-registration-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtwb\" (UniqueName: \"kubernetes.io/projected/4474ff46-c83c-4129-aee8-4109818d6fd7-kube-api-access-qqtwb\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-registration-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-modprobe-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.529954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-lib-modules\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.529985    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-daemon-config\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.530423 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-kubelet-config\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.530074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.030051157 +0000 UTC m=+2.068850284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-modprobe-d\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-kubelet-config\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cni-binary-copy\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-lib-modules\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e0fb6-82c9-4ed3-9734-16bb78615874-konnectivity-ca\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-socket-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-multus-certs\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkk5n\" (UniqueName: \"kubernetes.io/projected/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-kube-api-access-rkk5n\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgg2\" (UniqueName: \"kubernetes.io/projected/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-kube-api-access-4hgg2\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c704c5a-edcd-423d-831d-2cef17a01ecd-host\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-multus\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-config\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.531281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovn-node-metrics-cert\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhq8\" (UniqueName: \"kubernetes.io/projected/d218c9f2-9a63-4429-84b7-67d6d88169b9-kube-api-access-sdhq8\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhm6j\" (UniqueName: \"kubernetes.io/projected/9ce33184-76d1-4fdc-a7b2-d979b3f81245-kube-api-access-dhm6j\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.530963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovnkube-script-lib\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-daemon-config\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-sys\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-multus-certs\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-sys\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-etc-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bqc\" (UniqueName: \"kubernetes.io/projected/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-kube-api-access-l5bqc\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-socket-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.531723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-multus\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c704c5a-edcd-423d-831d-2cef17a01ecd-host\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.532186 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-sys-fs\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-netns\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-system-cni-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5h8\" (UniqueName: \"kubernetes.io/projected/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-kube-api-access-dx5h8\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-log-socket\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-bin\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-sys-fs\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-system-cni-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-log-socket\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-bin\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-netns\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.533036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-kubernetes\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7sd\" (UniqueName: \"kubernetes.io/projected/9c704c5a-edcd-423d-831d-2cef17a01ecd-kube-api-access-rj7sd\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-systemd\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.532990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-etc-openvswitch\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533128    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-systemd\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b6e0fb6-82c9-4ed3-9734-16bb78615874-konnectivity-ca\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-tuned\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-cni-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-kubernetes\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-netns\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-ovn\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.533813 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d218c9f2-9a63-4429-84b7-67d6d88169b9-tmp-dir\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c704c5a-edcd-423d-831d-2cef17a01ecd-serviceca\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-run\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-kubelet\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-run-ovn\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-netd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-netns\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.533981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjss\" (UniqueName: \"kubernetes.io/projected/2b5a509a-af25-4b9b-a670-d8b9473b90d4-kube-api-access-xqjss\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-run\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.534459 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-kubelet\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d218c9f2-9a63-4429-84b7-67d6d88169b9-tmp-dir\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-systemd-units\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c704c5a-edcd-423d-831d-2cef17a01ecd-serviceca\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-node-log\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-device-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-cni-netd\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-systemd-units\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-tmp\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-node-log\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-host\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-host\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-bin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-conf-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-device-dir\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-env-overrides\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.534969 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-var-lib-cni-bin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d218c9f2-9a63-4429-84b7-67d6d88169b9-hosts-file\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-multus-conf-dir\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ce33184-76d1-4fdc-a7b2-d979b3f81245-iptables-alerter-script\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.534989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d218c9f2-9a63-4429-84b7-67d6d88169b9-hosts-file\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-var-lib-kubelet\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4474ff46-c83c-4129-aee8-4109818d6fd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-k8s-cni-cncf-io\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-dbus\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-var-lib-kubelet\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b6e0fb6-82c9-4ed3-9734-16bb78615874-agent-certs\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2b5a509a-af25-4b9b-a670-d8b9473b90d4-env-overrides\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-conf\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cnibin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-slash\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.536882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ce33184-76d1-4fdc-a7b2-d979b3f81245-iptables-alerter-script\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b5a509a-af25-4b9b-a670-d8b9473b90d4-host-slash\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-cnibin\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d4820864-53d4-4b54-82dd-996b2be796ec-dbus\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-sysctl-conf\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.535895    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.535975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-host-run-k8s-cni-cncf-io\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.537762 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.536004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.035978112 +0000 UTC m=+2.074777228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:15.538135 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.537957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-etc-tuned\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.538135 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.538099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-tmp\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.538293 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.538273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b5a509a-af25-4b9b-a670-d8b9473b90d4-ovn-node-metrics-cert\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.538615 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.538579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b6e0fb6-82c9-4ed3-9734-16bb78615874-agent-certs\") pod \"konnectivity-agent-rc64q\" (UID: \"7b6e0fb6-82c9-4ed3-9734-16bb78615874\") " pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.540746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.540717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtwb\" (UniqueName: \"kubernetes.io/projected/4474ff46-c83c-4129-aee8-4109818d6fd7-kube-api-access-qqtwb\") pod \"aws-ebs-csi-driver-node-dmp4z\" (UID: \"4474ff46-c83c-4129-aee8-4109818d6fd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.541709 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.541686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bqc\" (UniqueName: \"kubernetes.io/projected/4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773-kube-api-access-l5bqc\") pod \"multus-additional-cni-plugins-9qvkp\" (UID: \"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773\") " pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.541797 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.541716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhq8\" (UniqueName: \"kubernetes.io/projected/d218c9f2-9a63-4429-84b7-67d6d88169b9-kube-api-access-sdhq8\") pod \"node-resolver-c9wd8\" (UID: \"d218c9f2-9a63-4429-84b7-67d6d88169b9\") " pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.542578 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.542553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkk5n\" (UniqueName: \"kubernetes.io/projected/59949b8f-d8c8-4681-a3a6-702ad0cadd1a-kube-api-access-rkk5n\") pod \"tuned-gg5w4\" (UID: \"59949b8f-d8c8-4681-a3a6-702ad0cadd1a\") " pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.542702 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.542663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhm6j\" (UniqueName: \"kubernetes.io/projected/9ce33184-76d1-4fdc-a7b2-d979b3f81245-kube-api-access-dhm6j\") pod \"iptables-alerter-fkq62\" (UID: \"9ce33184-76d1-4fdc-a7b2-d979b3f81245\") " pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.545844 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.545825    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:15.545844 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.545845    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:15.545956 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.545855    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:15.545956 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.545902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.045888333 +0000 UTC m=+2.084687435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:15.547015 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.546987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7sd\" (UniqueName: \"kubernetes.io/projected/9c704c5a-edcd-423d-831d-2cef17a01ecd-kube-api-access-rj7sd\") pod \"node-ca-8t554\" (UID: \"9c704c5a-edcd-423d-831d-2cef17a01ecd\") " pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.548843 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.548819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5h8\" (UniqueName: \"kubernetes.io/projected/0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45-kube-api-access-dx5h8\") pod \"multus-mmr78\" (UID: \"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45\") " pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.549326 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.549302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjss\" (UniqueName: \"kubernetes.io/projected/2b5a509a-af25-4b9b-a670-d8b9473b90d4-kube-api-access-xqjss\") pod \"ovnkube-node-g7j85\" (UID: \"2b5a509a-af25-4b9b-a670-d8b9473b90d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.549527 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.549510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgg2\" (UniqueName: \"kubernetes.io/projected/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-kube-api-access-4hgg2\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:15.594931 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.594890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e982e11182de362b0fd46b40e96f40f.slice/crio-df9665df167658507e3b92b86ca6b6c6731fdac75e9b8ff9a12f9c7fb6266089 WatchSource:0}: Error finding container df9665df167658507e3b92b86ca6b6c6731fdac75e9b8ff9a12f9c7fb6266089: Status 404 returned error can't find the container with id df9665df167658507e3b92b86ca6b6c6731fdac75e9b8ff9a12f9c7fb6266089
Sep 16 12:48:15.595270 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.595252    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfe345cfe49aded3ae47813e63145e8.slice/crio-9694773e65ee3907d40c4f2ba60ebbbd07aaa4fdf36093b1e13df31023c65649 WatchSource:0}: Error finding container 9694773e65ee3907d40c4f2ba60ebbbd07aaa4fdf36093b1e13df31023c65649: Status 404 returned error can't find the container with id 9694773e65ee3907d40c4f2ba60ebbbd07aaa4fdf36093b1e13df31023c65649
Sep 16 12:48:15.600072 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.600054    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:48:15.651354 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.651328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkq62"
Sep 16 12:48:15.661929 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.661908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce33184_76d1_4fdc_a7b2_d979b3f81245.slice/crio-fca5ea574b99bd44d052f4030fde18308c908829de8ca5fb1af8ab28ac9a61be WatchSource:0}: Error finding container fca5ea574b99bd44d052f4030fde18308c908829de8ca5fb1af8ab28ac9a61be: Status 404 returned error can't find the container with id fca5ea574b99bd44d052f4030fde18308c908829de8ca5fb1af8ab28ac9a61be
Sep 16 12:48:15.669748 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.669728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gg5w4"
Sep 16 12:48:15.677940 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.677911    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59949b8f_d8c8_4681_a3a6_702ad0cadd1a.slice/crio-4e372bcb29594382ed47096ce33b5ea64cc103268595dc18e350de05a7852541 WatchSource:0}: Error finding container 4e372bcb29594382ed47096ce33b5ea64cc103268595dc18e350de05a7852541: Status 404 returned error can't find the container with id 4e372bcb29594382ed47096ce33b5ea64cc103268595dc18e350de05a7852541
Sep 16 12:48:15.694785 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.694767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c9wd8"
Sep 16 12:48:15.703310 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.703291    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd218c9f2_9a63_4429_84b7_67d6d88169b9.slice/crio-6727e6b7c3ec66a7dd3ecc7d13f05644afeb609fa03cbabb10bae714a46e3f64 WatchSource:0}: Error finding container 6727e6b7c3ec66a7dd3ecc7d13f05644afeb609fa03cbabb10bae714a46e3f64: Status 404 returned error can't find the container with id 6727e6b7c3ec66a7dd3ecc7d13f05644afeb609fa03cbabb10bae714a46e3f64
Sep 16 12:48:15.714398 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.714377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z"
Sep 16 12:48:15.722665 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.722626    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474ff46_c83c_4129_aee8_4109818d6fd7.slice/crio-656775ccbd837c5fb3b0d83df585bc75eb8a8fb315627661f86e8cab95f52416 WatchSource:0}: Error finding container 656775ccbd837c5fb3b0d83df585bc75eb8a8fb315627661f86e8cab95f52416: Status 404 returned error can't find the container with id 656775ccbd837c5fb3b0d83df585bc75eb8a8fb315627661f86e8cab95f52416
Sep 16 12:48:15.725149 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.725131    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:15.728008 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.727988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mmr78"
Sep 16 12:48:15.735255 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.735237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:15.735933 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.735911    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8cc591_1a31_41e9_b2e8_3f8e66aa2d45.slice/crio-d43914bec00ad5248dd81f3297d18ecc83c3d80d3a3f27a2a018dfeb05e9371a WatchSource:0}: Error finding container d43914bec00ad5248dd81f3297d18ecc83c3d80d3a3f27a2a018dfeb05e9371a: Status 404 returned error can't find the container with id d43914bec00ad5248dd81f3297d18ecc83c3d80d3a3f27a2a018dfeb05e9371a
Sep 16 12:48:15.742955 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.742934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6e0fb6_82c9_4ed3_9734_16bb78615874.slice/crio-b3ea0841393e880f76602f3f92e6620f8016bcd1d289b13374973933375c782e WatchSource:0}: Error finding container b3ea0841393e880f76602f3f92e6620f8016bcd1d289b13374973933375c782e: Status 404 returned error can't find the container with id b3ea0841393e880f76602f3f92e6620f8016bcd1d289b13374973933375c782e
Sep 16 12:48:15.751137 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.751120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:15.759707 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.759459    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a509a_af25_4b9b_a670_d8b9473b90d4.slice/crio-e9abe47044b7dfbc3de3bbf4c64f745d81ad4b672b16a534d8c51191b7061526 WatchSource:0}: Error finding container e9abe47044b7dfbc3de3bbf4c64f745d81ad4b672b16a534d8c51191b7061526: Status 404 returned error can't find the container with id e9abe47044b7dfbc3de3bbf4c64f745d81ad4b672b16a534d8c51191b7061526
Sep 16 12:48:15.768357 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.768336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8t554"
Sep 16 12:48:15.774507 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:15.774490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9qvkp"
Sep 16 12:48:15.776751 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.776727    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c704c5a_edcd_423d_831d_2cef17a01ecd.slice/crio-79fb8fe6b0deecc6a49ed7c0d16c6e25bd123d8891e68219282109c9d0b7a254 WatchSource:0}: Error finding container 79fb8fe6b0deecc6a49ed7c0d16c6e25bd123d8891e68219282109c9d0b7a254: Status 404 returned error can't find the container with id 79fb8fe6b0deecc6a49ed7c0d16c6e25bd123d8891e68219282109c9d0b7a254
Sep 16 12:48:15.782982 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:15.782962    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8e3fb1_6e8f_4c8a_bb1a_15a59a9b0773.slice/crio-97f115ca2e9ff95067a80cc5c250e1e102b77ece342259a23e11b7152288ffe6 WatchSource:0}: Error finding container 97f115ca2e9ff95067a80cc5c250e1e102b77ece342259a23e11b7152288ffe6: Status 404 returned error can't find the container with id 97f115ca2e9ff95067a80cc5c250e1e102b77ece342259a23e11b7152288ffe6
Sep 16 12:48:15.785453 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.785434    2579 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-9qvkp_openshift-multus(4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 12:48:15.785507 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:15.785468    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" podUID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773"
Sep 16 12:48:16.038253 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.038154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:16.038253 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.038217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:16.038467 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.038332    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:16.038467 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.038348    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:16.038467 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.038397    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:17.038376807 +0000 UTC m=+3.077175909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:16.038467 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.038416    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:17.038407508 +0000 UTC m=+3.077206616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:16.139621 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.139568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:16.139813 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.139743    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:16.139813 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.139764    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:16.139813 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.139777    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:16.139953 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.139833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:17.139815078 +0000 UTC m=+3.178614184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:16.276561 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.276522    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:16.389750 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.389652    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:15 +0000 UTC" deadline="2028-06-11 07:32:35.501541146 +0000 UTC"
Sep 16 12:48:16.389750 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.389700    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15210h44m19.111846049s"
Sep 16 12:48:16.470377 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.469490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerStarted","Data":"97f115ca2e9ff95067a80cc5c250e1e102b77ece342259a23e11b7152288ffe6"}
Sep 16 12:48:16.477328 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:16.477295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:62c7cc09ff9f9684116e4c6b702d22ac67be29319eb961485e46f9441db87fb0\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" podUID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773"
Sep 16 12:48:16.480095 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.480017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8t554" event={"ID":"9c704c5a-edcd-423d-831d-2cef17a01ecd","Type":"ContainerStarted","Data":"79fb8fe6b0deecc6a49ed7c0d16c6e25bd123d8891e68219282109c9d0b7a254"}
Sep 16 12:48:16.502420 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.502379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rc64q" event={"ID":"7b6e0fb6-82c9-4ed3-9734-16bb78615874","Type":"ContainerStarted","Data":"b3ea0841393e880f76602f3f92e6620f8016bcd1d289b13374973933375c782e"}
Sep 16 12:48:16.507756 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.507721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c9wd8" event={"ID":"d218c9f2-9a63-4429-84b7-67d6d88169b9","Type":"ContainerStarted","Data":"6727e6b7c3ec66a7dd3ecc7d13f05644afeb609fa03cbabb10bae714a46e3f64"}
Sep 16 12:48:16.522848 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.522767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gg5w4" event={"ID":"59949b8f-d8c8-4681-a3a6-702ad0cadd1a","Type":"ContainerStarted","Data":"4e372bcb29594382ed47096ce33b5ea64cc103268595dc18e350de05a7852541"}
Sep 16 12:48:16.531992 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.531877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal" event={"ID":"7e982e11182de362b0fd46b40e96f40f","Type":"ContainerStarted","Data":"df9665df167658507e3b92b86ca6b6c6731fdac75e9b8ff9a12f9c7fb6266089"}
Sep 16 12:48:16.550941 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.550871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"e9abe47044b7dfbc3de3bbf4c64f745d81ad4b672b16a534d8c51191b7061526"}
Sep 16 12:48:16.554021 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.553987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmr78" event={"ID":"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45","Type":"ContainerStarted","Data":"d43914bec00ad5248dd81f3297d18ecc83c3d80d3a3f27a2a018dfeb05e9371a"}
Sep 16 12:48:16.567651 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.567600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z" event={"ID":"4474ff46-c83c-4129-aee8-4109818d6fd7","Type":"ContainerStarted","Data":"656775ccbd837c5fb3b0d83df585bc75eb8a8fb315627661f86e8cab95f52416"}
Sep 16 12:48:16.579886 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.579561    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:16.593148 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.593054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkq62" event={"ID":"9ce33184-76d1-4fdc-a7b2-d979b3f81245","Type":"ContainerStarted","Data":"fca5ea574b99bd44d052f4030fde18308c908829de8ca5fb1af8ab28ac9a61be"}
Sep 16 12:48:16.602270 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.602021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal" event={"ID":"3bfe345cfe49aded3ae47813e63145e8","Type":"ContainerStarted","Data":"9694773e65ee3907d40c4f2ba60ebbbd07aaa4fdf36093b1e13df31023c65649"}
Sep 16 12:48:16.961060 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:16.961028    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:17.046521 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.046477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:17.046756 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.046541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:17.046756 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.046728    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:17.046884 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.046797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:19.046776555 +0000 UTC m=+5.085575664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:17.046884 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.046865    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:17.047000 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.046896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:19.046885789 +0000 UTC m=+5.085684900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:17.147577 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.147451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:17.147785 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.147689    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:17.147785 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.147718    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:17.147785 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.147734    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:17.147963 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.147798    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:19.147778602 +0000 UTC m=+5.186577719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:17.390306 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.390189    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:15 +0000 UTC" deadline="2028-03-04 06:39:29.744380775 +0000 UTC"
Sep 16 12:48:17.390306 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.390254    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12833h51m12.354131132s"
Sep 16 12:48:17.453058 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.453015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:17.453240 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.453211    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:17.453659 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.453610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:17.453754 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.453732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:17.453815 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:17.453795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:17.453895 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:17.453870    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:19.063374 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.063334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:19.063835 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.063399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:19.063835 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.063555    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:19.063835 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.063630    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:23.063612592 +0000 UTC m=+9.102411700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:19.064108 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.064023    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:19.064108 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.064074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:23.064059813 +0000 UTC m=+9.102858921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:19.164780 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.164058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:19.164780 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.164288    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:19.164780 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.164311    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:19.164780 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.164325    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:19.164780 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.164389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:23.16436779 +0000 UTC m=+9.203166897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:19.452526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.452492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:19.452720 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.452538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:19.452720 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.452654    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:19.452840 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.452721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:19.452840 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:19.452791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:19.452942 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:19.452895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:21.453664 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:21.453063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:21.453664 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:21.453209    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:21.453664 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:21.453607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:21.454251 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:21.453703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:21.454251 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:21.453607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:21.454251 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:21.453793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:23.095974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.095927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:23.096408 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.095989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:23.096408 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.096143    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:23.096408 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.096204    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.096184829 +0000 UTC m=+17.134983950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:23.096660 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.096629    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:23.096716 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.096693    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.096675658 +0000 UTC m=+17.135474773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:23.197535 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.197297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:23.197535 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.197486    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:23.197535 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.197507    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:23.197535 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.197522    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:23.197800 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.197596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:31.197577953 +0000 UTC m=+17.236377060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:23.452632 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.452597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:23.452816 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.452742    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:23.453140 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.453117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:23.453228 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.453210    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:23.453273 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:23.453253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:23.453333 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:23.453316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:25.452499 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:25.452467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:25.453033 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:25.452588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:25.453033 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:25.452606    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:25.453033 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:25.452622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:25.453033 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:25.452702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:25.453033 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:25.452780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:27.452417 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:27.452377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:27.452417 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:27.452411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:27.452961 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:27.452421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:27.452961 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:27.452514    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:27.452961 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:27.452647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:27.452961 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:27.452751    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:29.452576 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:29.452501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:29.452576 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:29.452526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:29.452576 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:29.452553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:29.453132 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:29.452655    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:29.453132 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:29.452738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:29.453132 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:29.452764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:31.156473 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.156435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:31.156946 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.156477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:31.156946 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.156606    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:31.156946 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.156618    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:31.156946 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.156666    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.156648662 +0000 UTC m=+33.195447780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:31.156946 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.156680    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.156674125 +0000 UTC m=+33.195473227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:31.257578 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.257531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:31.257745 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.257718    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:31.257745 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.257740    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:31.257837 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.257750    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:31.257837 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.257812    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:48:47.257793669 +0000 UTC m=+33.296592771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:31.452826 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.452742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:31.452826 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.452793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:31.453028 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:31.452859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:31.453028 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.452873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:31.453028 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.452981    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:31.453169 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:31.453061    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:33.452953 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:33.452912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:33.453372 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:33.452912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:33.453372 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:33.453033    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:33.453372 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:33.452917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:33.453372 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:33.453127    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:33.453372 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:33.453238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:34.680442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.680408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gg5w4" event={"ID":"59949b8f-d8c8-4681-a3a6-702ad0cadd1a","Type":"ContainerStarted","Data":"8929e2ddf3255c81cbf7755f0c1ac5c4a926736498d4c56dc35dc18bbc80d1a4"}
Sep 16 12:48:34.684770 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.684739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mmr78" event={"ID":"0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45","Type":"ContainerStarted","Data":"dd1b7a8cdd9e902a120cb3581f8ccc840b942f9ecc48ad7cfa96d5609fc29ca2"}
Sep 16 12:48:34.689583 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.688728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal" event={"ID":"3bfe345cfe49aded3ae47813e63145e8","Type":"ContainerStarted","Data":"8457a2b353142933cccabad25c5ce51e0ba55d8e685d2a0aae34b04e94d359be"}
Sep 16 12:48:34.736850 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.736139    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gg5w4" podStartSLOduration=2.514368678 podStartE2EDuration="20.736117603s" totalImagesPullingTime="18.221748925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.703419177 +0000 UTC m=+20.742218297" watchObservedRunningTime="2026-09-16 12:48:34.736117603 +0000 UTC m=+20.774916727"
Sep 16 12:48:34.736850 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.736340    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mmr78" podStartSLOduration=2.287020792 podStartE2EDuration="20.736329467s" totalImagesPullingTime="18.449308675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.736306611 +0000 UTC m=+20.775105725" watchObservedRunningTime="2026-09-16 12:48:34.736329467 +0000 UTC m=+20.775128592"
Sep 16 12:48:34.758812 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:34.758757    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-128.ec2.internal" podStartSLOduration=19.758739893 podStartE2EDuration="19.758739893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:34.758429224 +0000 UTC m=+20.797228349" watchObservedRunningTime="2026-09-16 12:48:34.758739893 +0000 UTC m=+20.797539016"
Sep 16 12:48:35.453100 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.452843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:35.453301 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.453058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:35.453301 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:35.453214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:35.453301 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.453240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:35.453451 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:35.453354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:35.453451 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:35.453438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:35.692810 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.692779    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e982e11182de362b0fd46b40e96f40f" containerID="c873ec42d2fcd5fed870bc1c03b7280ac7014220dac128d59ae06325d6bd1c5f" exitCode=0
Sep 16 12:48:35.693365 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.692833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal" event={"ID":"7e982e11182de362b0fd46b40e96f40f","Type":"ContainerDied","Data":"c873ec42d2fcd5fed870bc1c03b7280ac7014220dac128d59ae06325d6bd1c5f"}
Sep 16 12:48:35.695792 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"8fea3d323c1d4f67b479824b28258fd12c5274813bf511850bf9700ea3ce4f63"}
Sep 16 12:48:35.695893 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"77b48df17c8b968258558e35f17cf868ea6a4a4cfa69f405b1e02bad3cf32fc7"}
Sep 16 12:48:35.695893 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"97f0f935472b4a3d1107b5567c65c2b0aeaa374fa83e7a802722eac98366188d"}
Sep 16 12:48:35.695963 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"c3196a85b1d249096a83f398906e686d34ae088cab1306bf5514a941036b44b3"}
Sep 16 12:48:35.695963 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"39c31aa4eb9d4d83a92bf97e3bd4cffe4ea4c566867beffc541ccbec3ecdb2ca"}
Sep 16 12:48:35.696029 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.695978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"a520aa666320e4f7c564a45ad7069d68b1b1344c2dc8bf1bd4f10ff5617fdc43"}
Sep 16 12:48:35.697158 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.697138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z" event={"ID":"4474ff46-c83c-4129-aee8-4109818d6fd7","Type":"ContainerStarted","Data":"6be698451ad985015a0383e5c4b610b3729a6ae58e009a9d9abfcff893806b25"}
Sep 16 12:48:35.698538 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.698517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkq62" event={"ID":"9ce33184-76d1-4fdc-a7b2-d979b3f81245","Type":"ContainerStarted","Data":"2cd2ba3b878b393ab2803a0d79bd770d082cbcba0fd762a48f646c0587ff9d24"}
Sep 16 12:48:35.699784 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.699767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8t554" event={"ID":"9c704c5a-edcd-423d-831d-2cef17a01ecd","Type":"ContainerStarted","Data":"dc4ffd36f422b3dc62cfa14336dec00b9a530dae4a0aea5b344e7cda48ccc8d7"}
Sep 16 12:48:35.701050 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.701030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rc64q" event={"ID":"7b6e0fb6-82c9-4ed3-9734-16bb78615874","Type":"ContainerStarted","Data":"bf35742e4a8a2204a2d968be9b4413c0e1a974029e68b267731e7009fff6aa8f"}
Sep 16 12:48:35.702238 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.702220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c9wd8" event={"ID":"d218c9f2-9a63-4429-84b7-67d6d88169b9","Type":"ContainerStarted","Data":"af5df5e0a6ae9f6b3537b476643b2e143eaba70e2af42e074e5c89b28f669e53"}
Sep 16 12:48:35.781939 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.781886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rc64q" podStartSLOduration=3.595786287 podStartE2EDuration="21.781867119s" totalImagesPullingTime="18.186080832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:35.78153442 +0000 UTC m=+21.820333543" watchObservedRunningTime="2026-09-16 12:48:35.781867119 +0000 UTC m=+21.820666242"
Sep 16 12:48:35.782420 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:35.782386    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c9wd8" podStartSLOduration=3.586534674 podStartE2EDuration="21.78237445s" totalImagesPullingTime="18.195839776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:35.756848551 +0000 UTC m=+21.795647675" watchObservedRunningTime="2026-09-16 12:48:35.78237445 +0000 UTC m=+21.821173573"
Sep 16 12:48:36.543939 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.543918    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:48:36.706278 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.706189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z" event={"ID":"4474ff46-c83c-4129-aee8-4109818d6fd7","Type":"ContainerStarted","Data":"f602962aa34f6d75d7a9aef9163d98b8facd01e97e5bec8d0d9738c9182676a4"}
Sep 16 12:48:36.707865 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.707837    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="63fd6eb8c4b7f5b0610c9bea79365f884d47458aae2c22bf57dcacf8eecb7aab" exitCode=0
Sep 16 12:48:36.707989 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.707907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"63fd6eb8c4b7f5b0610c9bea79365f884d47458aae2c22bf57dcacf8eecb7aab"}
Sep 16 12:48:36.709802 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.709771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal" event={"ID":"7e982e11182de362b0fd46b40e96f40f","Type":"ContainerStarted","Data":"4d98a8628080af9130236eea0ce6db7a6fad515daff292e05c188db9fa0769b8"}
Sep 16 12:48:36.735593 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.735535    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8t554" podStartSLOduration=4.582964198 podStartE2EDuration="22.735521285s" totalImagesPullingTime="18.152557087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:35.802114716 +0000 UTC m=+21.840913839" watchObservedRunningTime="2026-09-16 12:48:36.735521285 +0000 UTC m=+22.774320408"
Sep 16 12:48:36.759526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.759481    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-128.ec2.internal" podStartSLOduration=21.759466689 podStartE2EDuration="21.759466689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:36.758784128 +0000 UTC m=+22.797583250" watchObservedRunningTime="2026-09-16 12:48:36.759466689 +0000 UTC m=+22.798265811"
Sep 16 12:48:36.781424 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:36.781383    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fkq62" podStartSLOduration=4.510514127 podStartE2EDuration="22.781370365s" totalImagesPullingTime="18.270856238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:36.781094994 +0000 UTC m=+22.819894116" watchObservedRunningTime="2026-09-16 12:48:36.781370365 +0000 UTC m=+22.820169487"
Sep 16 12:48:37.385791 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.385462    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:36.543934619Z","UUID":"1c15c194-1890-49b4-aa39-bb27bff0f069","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:48:37.387488 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.387466    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:48:37.387630 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.387499    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:48:37.453062 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.452980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:37.453208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.452997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:37.453208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.452997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:37.453208 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:37.453142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:37.453342 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:37.453221    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:37.453342 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:37.453297    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:37.715219 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.715134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z" event={"ID":"4474ff46-c83c-4129-aee8-4109818d6fd7","Type":"ContainerStarted","Data":"4b80f60dd97b5253f1372923350a39984c24e9b53b9984b24925bc9fc4eecc51"}
Sep 16 12:48:37.719140 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:37.719100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"21ec23d97ec076d915ff5a573e062fac0cbb73eff4d7af28ae86b41895ffdb24"}
Sep 16 12:48:38.869964 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:38.869927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:38.870608 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:38.870582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:38.888808 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:38.888764    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmp4z" podStartSLOduration=4.453079388 podStartE2EDuration="24.888747793s" totalImagesPullingTime="20.435668405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:37.737146296 +0000 UTC m=+23.775945419" watchObservedRunningTime="2026-09-16 12:48:38.888747793 +0000 UTC m=+24.927546915"
Sep 16 12:48:39.452640 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:39.452607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:39.452804 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:39.452605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:39.452804 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:39.452751    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:39.452804 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:39.452608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:39.452934 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:39.452790    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:39.452934 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:39.452875    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:39.723640 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:39.723613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:39.724240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:39.724217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rc64q"
Sep 16 12:48:41.452383 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.452165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:41.452849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.452173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:41.452849 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:41.452420    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:41.452849 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:41.452522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:41.452849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.452185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:41.452849 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:41.452646    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:41.728913 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.728840    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="b500edd887ccd4fa2a827735872d83a2d737d9abc22834a97b2104a51ab69b16" exitCode=0
Sep 16 12:48:41.729076 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.728909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"b500edd887ccd4fa2a827735872d83a2d737d9abc22834a97b2104a51ab69b16"}
Sep 16 12:48:41.732860 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.732841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" event={"ID":"2b5a509a-af25-4b9b-a670-d8b9473b90d4","Type":"ContainerStarted","Data":"4b9d332f44152a9ed4342ca2e9a584523db34e70457c4c4fc7482a6dc8f977fe"}
Sep 16 12:48:41.733181 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.733160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:41.733283 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.733224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:41.748067 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.748046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:41.816096 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:41.816040    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85" podStartSLOduration=8.930280463 podStartE2EDuration="27.816023985s" totalImagesPullingTime="18.885743522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:41.814389601 +0000 UTC m=+27.853188725" watchObservedRunningTime="2026-09-16 12:48:41.816023985 +0000 UTC m=+27.854823108"
Sep 16 12:48:42.738011 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.737979    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="8b977024e6015e001541543af706811f71dd0002a5ad8a25b37f21ec2ddf8d9f" exitCode=0
Sep 16 12:48:42.738615 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.738047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"8b977024e6015e001541543af706811f71dd0002a5ad8a25b37f21ec2ddf8d9f"}
Sep 16 12:48:42.738693 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.738617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:42.754508 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.754478    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:48:42.951271 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.951036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p2tq4"]
Sep 16 12:48:42.951450 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.951372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:42.951520 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:42.951484    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:42.951719 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.951694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57zxt"]
Sep 16 12:48:42.951909 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.951807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:42.951981 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:42.951925    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:42.952400 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.952382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-74lv5"]
Sep 16 12:48:42.952490 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:42.952478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:42.952613 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:42.952589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:43.743584 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:43.743539    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="d7ac369bb8794b06bbb152dadf5026a98ea783ccf72de2ce2c6866f126082682" exitCode=0
Sep 16 12:48:43.743974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:43.743598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"d7ac369bb8794b06bbb152dadf5026a98ea783ccf72de2ce2c6866f126082682"}
Sep 16 12:48:44.454294 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:44.454184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:44.454294 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:44.454283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:44.454496 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:44.454321    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:44.454496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:44.454326    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:44.454496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:44.454394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:44.454496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:44.454478    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:46.452154 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:46.452117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:46.452722 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:46.452115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:46.452722 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:46.452255    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74lv5" podUID="d4820864-53d4-4b54-82dd-996b2be796ec"
Sep 16 12:48:46.452722 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:46.452132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:46.452722 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:46.452348    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p2tq4" podUID="3a0399ab-d7ad-410e-9383-8e8eaaeb106e"
Sep 16 12:48:46.452722 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:46.452460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57zxt" podUID="5487dc67-8a7e-4ff9-8a18-85d8f102b01e"
Sep 16 12:48:47.168963 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.168919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:47.169152 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.169027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:47.169152 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.169097    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:47.169152 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.169139    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:47.169311 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.169200    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret podName:d4820864-53d4-4b54-82dd-996b2be796ec nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.169179324 +0000 UTC m=+65.207978428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret") pod "global-pull-secret-syncer-74lv5" (UID: "d4820864-53d4-4b54-82dd-996b2be796ec") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:47.169311 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.169218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.169207882 +0000 UTC m=+65.208006992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:47.227763 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.227694    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-128.ec2.internal" event="NodeReady"
Sep 16 12:48:47.227921 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.227856    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:48:47.270317 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.270281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:47.270496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.270420    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:47.270496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.270449    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:47.270496 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.270463    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qc7kg for pod openshift-network-diagnostics/network-check-target-p2tq4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:47.270661 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.270519    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg podName:3a0399ab-d7ad-410e-9383-8e8eaaeb106e nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.270500393 +0000 UTC m=+65.309299510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qc7kg" (UniqueName: "kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg") pod "network-check-target-p2tq4" (UID: "3a0399ab-d7ad-410e-9383-8e8eaaeb106e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:47.314538 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.314502    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jldt8"]
Sep 16 12:48:47.355873 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.355835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cm276"]
Sep 16 12:48:47.356031 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.355981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.360308 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.360277    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:48:47.360481 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.360405    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:48:47.360481 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.360430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:48:47.360685 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.360609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t2wwc\""
Sep 16 12:48:47.377921 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.377890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jldt8"]
Sep 16 12:48:47.377921 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.377930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cm276"]
Sep 16 12:48:47.378143 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.378020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.382817 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.382790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:48:47.382951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.382823    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:48:47.382951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.382867    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwp6v\""
Sep 16 12:48:47.472153 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.472595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bcc27-8289-4ce0-896f-f530c2a30fba-tmp-dir\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.472595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzp4\" (UniqueName: \"kubernetes.io/projected/a07394e1-5dc3-4faa-a8ec-517ab84483e2-kube-api-access-8rzp4\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.472595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r79l\" (UniqueName: \"kubernetes.io/projected/0c3bcc27-8289-4ce0-896f-f530c2a30fba-kube-api-access-7r79l\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.472595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3bcc27-8289-4ce0-896f-f530c2a30fba-config-volume\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.472595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.472382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a07394e1-5dc3-4faa-a8ec-517ab84483e2-cert\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.573170 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3bcc27-8289-4ce0-896f-f530c2a30fba-config-volume\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.573170 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a07394e1-5dc3-4faa-a8ec-517ab84483e2-cert\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.573387 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.573387 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bcc27-8289-4ce0-896f-f530c2a30fba-tmp-dir\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.573387 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.573328    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:47.573529 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:47.573404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:48:48.07337991 +0000 UTC m=+34.112179017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:47.573625 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bcc27-8289-4ce0-896f-f530c2a30fba-tmp-dir\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.573697 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzp4\" (UniqueName: \"kubernetes.io/projected/a07394e1-5dc3-4faa-a8ec-517ab84483e2-kube-api-access-8rzp4\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.573697 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r79l\" (UniqueName: \"kubernetes.io/projected/0c3bcc27-8289-4ce0-896f-f530c2a30fba-kube-api-access-7r79l\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.573697 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.573684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3bcc27-8289-4ce0-896f-f530c2a30fba-config-volume\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.578073 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.578045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a07394e1-5dc3-4faa-a8ec-517ab84483e2-cert\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.590263 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.590240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzp4\" (UniqueName: \"kubernetes.io/projected/a07394e1-5dc3-4faa-a8ec-517ab84483e2-kube-api-access-8rzp4\") pod \"ingress-canary-jldt8\" (UID: \"a07394e1-5dc3-4faa-a8ec-517ab84483e2\") " pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:47.597139 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.597114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r79l\" (UniqueName: \"kubernetes.io/projected/0c3bcc27-8289-4ce0-896f-f530c2a30fba-kube-api-access-7r79l\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:47.668202 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:47.668166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jldt8"
Sep 16 12:48:48.077507 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.077326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:48.077692 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:48.077488    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:48.077692 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:48.077635    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:48:49.077618609 +0000 UTC m=+35.116417710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:48.228954 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.228875    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jldt8"]
Sep 16 12:48:48.233634 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:48:48.233610    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07394e1_5dc3_4faa_a8ec_517ab84483e2.slice/crio-48709198ed311a6d37b7bd64990b0d3f37f6a8e2959dd24d89b04eca5d2cf122 WatchSource:0}: Error finding container 48709198ed311a6d37b7bd64990b0d3f37f6a8e2959dd24d89b04eca5d2cf122: Status 404 returned error can't find the container with id 48709198ed311a6d37b7bd64990b0d3f37f6a8e2959dd24d89b04eca5d2cf122
Sep 16 12:48:48.452607 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.452579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:48:48.452731 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.452581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:48:48.452785 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.452744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:48:48.461209 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.461187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:48.462281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.462262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:48.462389 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.462264    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:48.462492 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.462473    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v25dx\""
Sep 16 12:48:48.462807 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.462787    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:48.463633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.463614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlkh7\""
Sep 16 12:48:48.756397 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.756348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jldt8" event={"ID":"a07394e1-5dc3-4faa-a8ec-517ab84483e2","Type":"ContainerStarted","Data":"48709198ed311a6d37b7bd64990b0d3f37f6a8e2959dd24d89b04eca5d2cf122"}
Sep 16 12:48:48.759670 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.759641    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="bc0e74aa9d6861c39a9818ee75404c16ae0679cc6a01c0c3646123bacc5465e4" exitCode=0
Sep 16 12:48:48.759804 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:48.759685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"bc0e74aa9d6861c39a9818ee75404c16ae0679cc6a01c0c3646123bacc5465e4"}
Sep 16 12:48:49.084829 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:49.084753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:49.084997 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:49.084884    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:49.084997 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:49.084953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:48:51.084935691 +0000 UTC m=+37.123734797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:49.766533 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:49.766497    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773" containerID="cdc831a4851892f2ed89e96ef42a8a8828acb18853b39be8a592cf1ad65f6747" exitCode=0
Sep 16 12:48:49.767117 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:49.766600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerDied","Data":"cdc831a4851892f2ed89e96ef42a8a8828acb18853b39be8a592cf1ad65f6747"}
Sep 16 12:48:50.770180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:50.770087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jldt8" event={"ID":"a07394e1-5dc3-4faa-a8ec-517ab84483e2","Type":"ContainerStarted","Data":"e7f104ee4f3e27ee620fd5a81c0ed02e96091bc373abbaa7e176aaa9f76c0c48"}
Sep 16 12:48:50.773749 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:50.773720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" event={"ID":"4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773","Type":"ContainerStarted","Data":"b436516bf1c8c3b4da4fa21cacb92b1bcbe399f3d5967932f5dd303e261a44c8"}
Sep 16 12:48:50.789425 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:50.789376    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jldt8" podStartSLOduration=1.431855542 podStartE2EDuration="3.789361717s" totalImagesPullingTime="2.357506175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.788137702 +0000 UTC m=+36.826936825" watchObservedRunningTime="2026-09-16 12:48:50.789361717 +0000 UTC m=+36.828160839"
Sep 16 12:48:50.821095 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:50.821038    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9qvkp" podStartSLOduration=4.369412544 podStartE2EDuration="36.821013513s" totalImagesPullingTime="32.451600969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-16 12:48:50.820846763 +0000 UTC m=+36.859645899" watchObservedRunningTime="2026-09-16 12:48:50.821013513 +0000 UTC m=+36.859812637"
Sep 16 12:48:51.104309 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:51.104225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:51.104448 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:51.104349    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:51.104448 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:51.104400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:48:55.104386972 +0000 UTC m=+41.143186073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:55.133502 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:48:55.133318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:48:55.133883 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:55.133459    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:55.133883 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:48:55.133589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:49:03.133573679 +0000 UTC m=+49.172372793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:49:03.188244 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:03.188191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:03.188768 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:03.188362    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:49:03.188768 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:03.188446    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:49:19.188424035 +0000 UTC m=+65.227223152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:49:11.000141 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.000109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ljz6"]
Sep 16 12:49:11.004166 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.004143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.010126 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.010083    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 12:49:11.010922 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.010904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 12:49:11.011427 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.011412    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.011660 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.011640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fr7gq\""
Sep 16 12:49:11.011734 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.011640    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.016265 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.016247    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 12:49:11.021912 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.021893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ljz6"]
Sep 16 12:49:11.108745 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.108713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55fccc44b8-9v5dg"]
Sep 16 12:49:11.112307 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.112288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.118039 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.117876    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.118113 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118088    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-chr7n\""
Sep 16 12:49:11.118380 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 12:49:11.118662 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 12:49:11.118746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118663    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 12:49:11.118746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 12:49:11.118746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.118711    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.128133 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.128112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55fccc44b8-9v5dg"]
Sep 16 12:49:11.142359 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.142338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmtzz\" (UniqueName: \"kubernetes.io/projected/92004b1b-d2d8-413a-be62-9def7a380328-kube-api-access-zmtzz\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.142443 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.142377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92004b1b-d2d8-413a-be62-9def7a380328-serving-cert\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.142522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.142500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-config\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.142648 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.142530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-trusted-ca\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.243927 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.243882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-default-certificate\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.243927 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.243927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-config\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.244180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.243942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-trusted-ca\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.244180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-stats-auth\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.244180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmtzz\" (UniqueName: \"kubernetes.io/projected/92004b1b-d2d8-413a-be62-9def7a380328-kube-api-access-zmtzz\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.244180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.244343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.244343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcg2\" (UniqueName: \"kubernetes.io/projected/c671399e-37d1-4307-8c61-a9464e26f327-kube-api-access-prcg2\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.244343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92004b1b-d2d8-413a-be62-9def7a380328-serving-cert\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.244757 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-config\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.244978 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.244953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92004b1b-d2d8-413a-be62-9def7a380328-trusted-ca\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.247514 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.247498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92004b1b-d2d8-413a-be62-9def7a380328-serving-cert\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.257903 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.257838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmtzz\" (UniqueName: \"kubernetes.io/projected/92004b1b-d2d8-413a-be62-9def7a380328-kube-api-access-zmtzz\") pod \"console-operator-696884fb9c-7ljz6\" (UID: \"92004b1b-d2d8-413a-be62-9def7a380328\") " pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.312808 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.312769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:11.344971 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.344936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-default-certificate\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.345111 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.344983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-stats-auth\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.345111 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.345011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.345231 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.345120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.345231 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.345138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:11.845118786 +0000 UTC m=+57.883917913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:11.345231 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.345169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcg2\" (UniqueName: \"kubernetes.io/projected/c671399e-37d1-4307-8c61-a9464e26f327-kube-api-access-prcg2\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.345231 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.345196    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:11.345435 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.345245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:11.845232702 +0000 UTC m=+57.884031803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : secret "router-metrics-certs-default" not found
Sep 16 12:49:11.347424 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.347401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-stats-auth\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.347516 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.347440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-default-certificate\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.360828 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.360804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcg2\" (UniqueName: \"kubernetes.io/projected/c671399e-37d1-4307-8c61-a9464e26f327-kube-api-access-prcg2\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.585997 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.585914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7ljz6"]
Sep 16 12:49:11.589100 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:11.589074    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92004b1b_d2d8_413a_be62_9def7a380328.slice/crio-1c4117cf68776f8e137846a9508a83499f435635158b858c681ad0860c3961a1 WatchSource:0}: Error finding container 1c4117cf68776f8e137846a9508a83499f435635158b858c681ad0860c3961a1: Status 404 returned error can't find the container with id 1c4117cf68776f8e137846a9508a83499f435635158b858c681ad0860c3961a1
Sep 16 12:49:11.823147 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.823113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" event={"ID":"92004b1b-d2d8-413a-be62-9def7a380328","Type":"ContainerStarted","Data":"1c4117cf68776f8e137846a9508a83499f435635158b858c681ad0860c3961a1"}
Sep 16 12:49:11.849544 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.849473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.849544 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:11.849505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:11.849718 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.849619    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:11.849718 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.849662    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:12.849643388 +0000 UTC m=+58.888442488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:11.849718 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:11.849681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:12.849673697 +0000 UTC m=+58.888472798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : secret "router-metrics-certs-default" not found
Sep 16 12:49:12.857648 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:12.857606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:12.858025 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:12.857658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:12.858025 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:12.857786    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:12.858025 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:12.857804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.857783719 +0000 UTC m=+60.896582837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:12.858025 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:12.857848    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.857835453 +0000 UTC m=+60.896634568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : secret "router-metrics-certs-default" not found
Sep 16 12:49:12.990132 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:12.990098    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"]
Sep 16 12:49:12.994753 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:12.994730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.000744 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.000721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 12:49:13.001065 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.001039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:49:13.001210 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.001178    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 12:49:13.002265 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.002240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 12:49:13.002370 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.002247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9d6sk\""
Sep 16 12:49:13.020534 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.020507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"]
Sep 16 12:49:13.159406 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.159370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b269419-28c4-4b80-80aa-50956087db1b-config\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.159586 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.159525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b269419-28c4-4b80-80aa-50956087db1b-serving-cert\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.159586 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.159553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7qm\" (UniqueName: \"kubernetes.io/projected/5b269419-28c4-4b80-80aa-50956087db1b-kube-api-access-ql7qm\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.260130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.260089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b269419-28c4-4b80-80aa-50956087db1b-serving-cert\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.260321 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.260140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7qm\" (UniqueName: \"kubernetes.io/projected/5b269419-28c4-4b80-80aa-50956087db1b-kube-api-access-ql7qm\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.260321 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.260196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b269419-28c4-4b80-80aa-50956087db1b-config\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.260926 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.260899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b269419-28c4-4b80-80aa-50956087db1b-config\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.262961 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.262934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5b269419-28c4-4b80-80aa-50956087db1b-serving-cert\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.270240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.270211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7qm\" (UniqueName: \"kubernetes.io/projected/5b269419-28c4-4b80-80aa-50956087db1b-kube-api-access-ql7qm\") pod \"service-ca-operator-cf7c776b7-9lf55\" (UID: \"5b269419-28c4-4b80-80aa-50956087db1b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.305409 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.305367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"
Sep 16 12:49:13.603497 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.603469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55"]
Sep 16 12:49:13.606668 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:13.606642    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b269419_28c4_4b80_80aa_50956087db1b.slice/crio-740ed3488a7a0712eda07cd8f7a6f93f2068f08a625b8466f47e2d0020346817 WatchSource:0}: Error finding container 740ed3488a7a0712eda07cd8f7a6f93f2068f08a625b8466f47e2d0020346817: Status 404 returned error can't find the container with id 740ed3488a7a0712eda07cd8f7a6f93f2068f08a625b8466f47e2d0020346817
Sep 16 12:49:13.829639 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.829540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/0.log"
Sep 16 12:49:13.829639 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.829596    2579 generic.go:356] "Generic (PLEG): container finished" podID="92004b1b-d2d8-413a-be62-9def7a380328" containerID="292b95735017af80ef74289bf02148e008bb22d0af3136f270b8a248735e3843" exitCode=255
Sep 16 12:49:13.829639 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.829627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" event={"ID":"92004b1b-d2d8-413a-be62-9def7a380328","Type":"ContainerDied","Data":"292b95735017af80ef74289bf02148e008bb22d0af3136f270b8a248735e3843"}
Sep 16 12:49:13.829889 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.829873    2579 scope.go:117] "RemoveContainer" containerID="292b95735017af80ef74289bf02148e008bb22d0af3136f270b8a248735e3843"
Sep 16 12:49:13.830956 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:13.830935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55" event={"ID":"5b269419-28c4-4b80-80aa-50956087db1b","Type":"ContainerStarted","Data":"740ed3488a7a0712eda07cd8f7a6f93f2068f08a625b8466f47e2d0020346817"}
Sep 16 12:49:14.763983 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.763958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7j85"
Sep 16 12:49:14.835036 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:49:14.835451 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/0.log"
Sep 16 12:49:14.835556 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835473    2579 generic.go:356] "Generic (PLEG): container finished" podID="92004b1b-d2d8-413a-be62-9def7a380328" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc" exitCode=255
Sep 16 12:49:14.835556 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" event={"ID":"92004b1b-d2d8-413a-be62-9def7a380328","Type":"ContainerDied","Data":"cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"}
Sep 16 12:49:14.835683 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835634    2579 scope.go:117] "RemoveContainer" containerID="292b95735017af80ef74289bf02148e008bb22d0af3136f270b8a248735e3843"
Sep 16 12:49:14.835829 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.835810    2579 scope.go:117] "RemoveContainer" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"
Sep 16 12:49:14.836020 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:14.835993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ljz6_openshift-console-operator(92004b1b-d2d8-413a-be62-9def7a380328)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" podUID="92004b1b-d2d8-413a-be62-9def7a380328"
Sep 16 12:49:14.874867 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.874837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:14.875011 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:14.874877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:14.875086 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:14.875003    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.874980004 +0000 UTC m=+64.913779121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:14.875250 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:14.875226    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:14.875357 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:14.875284    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.875270263 +0000 UTC m=+64.914069364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : secret "router-metrics-certs-default" not found
Sep 16 12:49:15.839660 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:15.839577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:49:15.840103 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:15.839939    2579 scope.go:117] "RemoveContainer" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"
Sep 16 12:49:15.840156 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:15.840110    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ljz6_openshift-console-operator(92004b1b-d2d8-413a-be62-9def7a380328)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" podUID="92004b1b-d2d8-413a-be62-9def7a380328"
Sep 16 12:49:15.841176 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:15.841158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55" event={"ID":"5b269419-28c4-4b80-80aa-50956087db1b","Type":"ContainerStarted","Data":"2ee791b7d451db88f3cd84af627c984b02a95f7547e1cf2c8952d4b96c0c0dd9"}
Sep 16 12:49:15.889246 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:15.889185    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55" podStartSLOduration=1.944356964 podStartE2EDuration="3.889164001s" totalImagesPullingTime="1.944807037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:15.889095387 +0000 UTC m=+61.927894510" watchObservedRunningTime="2026-09-16 12:49:15.889164001 +0000 UTC m=+61.927963127"
Sep 16 12:49:16.976474 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.976438    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rcht6"]
Sep 16 12:49:16.978648 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.978632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:16.981873 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.981853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:49:16.981974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.981852    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:49:16.983032 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.983017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bq7jb\""
Sep 16 12:49:16.983439 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.983294    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:49:16.983439 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.983391    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:49:16.998443 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:16.998419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rcht6"]
Sep 16 12:49:17.093058 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.093018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.093217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.093079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.093217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.093132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-crio-socket\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.093217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.093181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd7w\" (UniqueName: \"kubernetes.io/projected/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-api-access-kdd7w\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.093217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.093204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-data-volume\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194222 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd7w\" (UniqueName: \"kubernetes.io/projected/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-api-access-kdd7w\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194222 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-data-volume\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194436 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194436 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194436 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-crio-socket\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194436 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-crio-socket\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.194436 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.194425    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.194613 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.194489    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:17.694473839 +0000 UTC m=+63.733272945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.194613 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.194422    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.194613 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.194596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:17.694582087 +0000 UTC m=+63.733381193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.194719 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.194660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-data-volume\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.205431 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.205403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd7w\" (UniqueName: \"kubernetes.io/projected/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-api-access-kdd7w\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.698971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:17.699069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.699236    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.699311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.69929078 +0000 UTC m=+64.738089883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.699831    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.701127 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:17.699891    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.699872837 +0000 UTC m=+64.738671938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:18.706730 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:18.706697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:18.707185 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:18.706756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:18.707185 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.706842    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:18.707185 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.706857    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:18.707185 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.706918    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.706898213 +0000 UTC m=+66.745697315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:18.707185 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.706938    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.706927441 +0000 UTC m=+66.745726543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:18.908068 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:18.908028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:18.908068 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:18.908071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:18.908272 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.908210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:26.908189186 +0000 UTC m=+72.946988312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:18.908323 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.908274    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 12:49:18.908356 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:18.908339    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:26.908321396 +0000 UTC m=+72.947120499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : secret "router-metrics-certs-default" not found
Sep 16 12:49:19.073524 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.073453    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9777t"]
Sep 16 12:49:19.075428 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.075413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.080950 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.080924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 12:49:19.080950 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.080937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 12:49:19.082088 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.082067    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 12:49:19.082194 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.082113    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 12:49:19.082244 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.082187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mwfj9\""
Sep 16 12:49:19.092866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.092838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9777t"]
Sep 16 12:49:19.210974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.210931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bb7\" (UniqueName: \"kubernetes.io/projected/25f6842a-d4a1-465e-a106-ee847a12c7b3-kube-api-access-n6bb7\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.210974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.210974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:49:19.211240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.211001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:49:19.211240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.211063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-key\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.211240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.211109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-cabundle\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.211240 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.211161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:19.211429 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:19.211280    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:49:19.211429 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:19.211367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls podName:0c3bcc27-8289-4ce0-896f-f530c2a30fba nodeName:}" failed. No retries permitted until 2026-09-16 12:49:51.211342334 +0000 UTC m=+97.250141454 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls") pod "dns-default-cm276" (UID: "0c3bcc27-8289-4ce0-896f-f530c2a30fba") : secret "dns-default-metrics-tls" not found
Sep 16 12:49:19.213838 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.213815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:49:19.213956 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.213855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:49:19.221867 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:19.221843    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:49:19.221938 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:19.221916    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs podName:5487dc67-8a7e-4ff9-8a18-85d8f102b01e nodeName:}" failed. No retries permitted until 2026-09-16 12:50:23.221895707 +0000 UTC m=+129.260694807 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs") pod "network-metrics-daemon-57zxt" (UID: "5487dc67-8a7e-4ff9-8a18-85d8f102b01e") : secret "metrics-daemon-secret" not found
Sep 16 12:49:19.224100 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.224079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d4820864-53d4-4b54-82dd-996b2be796ec-original-pull-secret\") pod \"global-pull-secret-syncer-74lv5\" (UID: \"d4820864-53d4-4b54-82dd-996b2be796ec\") " pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:49:19.311723 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.311680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bb7\" (UniqueName: \"kubernetes.io/projected/25f6842a-d4a1-465e-a106-ee847a12c7b3-kube-api-access-n6bb7\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.311930 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.311752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-key\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.311930 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.311777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-cabundle\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.311930 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.311805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:49:19.312428 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.312396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-cabundle\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.314212 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.314185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25f6842a-d4a1-465e-a106-ee847a12c7b3-signing-key\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.315357 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.315341    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:49:19.321143 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.321121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bb7\" (UniqueName: \"kubernetes.io/projected/25f6842a-d4a1-465e-a106-ee847a12c7b3-kube-api-access-n6bb7\") pod \"service-ca-568869fdc5-9777t\" (UID: \"25f6842a-d4a1-465e-a106-ee847a12c7b3\") " pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.325451 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.325391    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:49:19.335991 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.335962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7kg\" (UniqueName: \"kubernetes.io/projected/3a0399ab-d7ad-410e-9383-8e8eaaeb106e-kube-api-access-qc7kg\") pod \"network-check-target-p2tq4\" (UID: \"3a0399ab-d7ad-410e-9383-8e8eaaeb106e\") " pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:49:19.384642 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.384600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-9777t"
Sep 16 12:49:19.415670 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.415636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74lv5"
Sep 16 12:49:19.432793 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.432767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlkh7\""
Sep 16 12:49:19.439313 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.439025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:49:19.657479 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.657452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-9777t"]
Sep 16 12:49:19.660492 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:19.660471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f6842a_d4a1_465e_a106_ee847a12c7b3.slice/crio-36c23ac16ddc23c799a25cbf7875f4bc84b41c709c2bfa130914a29598107430 WatchSource:0}: Error finding container 36c23ac16ddc23c799a25cbf7875f4bc84b41c709c2bfa130914a29598107430: Status 404 returned error can't find the container with id 36c23ac16ddc23c799a25cbf7875f4bc84b41c709c2bfa130914a29598107430
Sep 16 12:49:19.692737 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.692712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c9wd8_d218c9f2-9a63-4429-84b7-67d6d88169b9/dns-node-resolver/0.log"
Sep 16 12:49:19.696007 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.695982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-74lv5"]
Sep 16 12:49:19.700195 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:19.700170    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4820864_53d4_4b54_82dd_996b2be796ec.slice/crio-fd54ef43c66588c870077f230fe027f7079073cbee0409276a6f98ca33fc0ce3 WatchSource:0}: Error finding container fd54ef43c66588c870077f230fe027f7079073cbee0409276a6f98ca33fc0ce3: Status 404 returned error can't find the container with id fd54ef43c66588c870077f230fe027f7079073cbee0409276a6f98ca33fc0ce3
Sep 16 12:49:19.715783 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.715759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p2tq4"]
Sep 16 12:49:19.718631 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:19.718604    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0399ab_d7ad_410e_9383_8e8eaaeb106e.slice/crio-822da6e450fa975a57f86fa50e7dc82934ce1a839fad7c0c1eff23c00a4b0986 WatchSource:0}: Error finding container 822da6e450fa975a57f86fa50e7dc82934ce1a839fad7c0c1eff23c00a4b0986: Status 404 returned error can't find the container with id 822da6e450fa975a57f86fa50e7dc82934ce1a839fad7c0c1eff23c00a4b0986
Sep 16 12:49:19.852788 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.852752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-74lv5" event={"ID":"d4820864-53d4-4b54-82dd-996b2be796ec","Type":"ContainerStarted","Data":"fd54ef43c66588c870077f230fe027f7079073cbee0409276a6f98ca33fc0ce3"}
Sep 16 12:49:19.854519 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.854490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9777t" event={"ID":"25f6842a-d4a1-465e-a106-ee847a12c7b3","Type":"ContainerStarted","Data":"552e5ae5f4fb74e223488a5216cab235bd43f219e6568d8ce8c1edc318f12de7"}
Sep 16 12:49:19.854686 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.854539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-9777t" event={"ID":"25f6842a-d4a1-465e-a106-ee847a12c7b3","Type":"ContainerStarted","Data":"36c23ac16ddc23c799a25cbf7875f4bc84b41c709c2bfa130914a29598107430"}
Sep 16 12:49:19.855734 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.855715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p2tq4" event={"ID":"3a0399ab-d7ad-410e-9383-8e8eaaeb106e","Type":"ContainerStarted","Data":"822da6e450fa975a57f86fa50e7dc82934ce1a839fad7c0c1eff23c00a4b0986"}
Sep 16 12:49:19.871753 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:19.871664    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-9777t" podStartSLOduration=0.87164896 podStartE2EDuration="871.64896ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:19.870710564 +0000 UTC m=+65.909509687" watchObservedRunningTime="2026-09-16 12:49:19.87164896 +0000 UTC m=+65.910448082"
Sep 16 12:49:20.488368 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:20.488326    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8t554_9c704c5a-edcd-423d-831d-2cef17a01ecd/node-ca/0.log"
Sep 16 12:49:20.723189 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:20.723134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:20.723699 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:20.723210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:20.723699 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:20.723325    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:20.723699 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:20.723342    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:20.723699 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:20.723398    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:24.723378818 +0000 UTC m=+70.762177933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:20.723699 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:20.723416    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:24.723407422 +0000 UTC m=+70.762206523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:21.288101 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:21.288035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jldt8_a07394e1-5dc3-4faa-a8ec-517ab84483e2/serve-healthcheck-canary/0.log"
Sep 16 12:49:21.313673 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:21.312950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:21.313673 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:21.313392    2579 scope.go:117] "RemoveContainer" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"
Sep 16 12:49:21.313673 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:21.313624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:21.314421 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:21.313556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ljz6_openshift-console-operator(92004b1b-d2d8-413a-be62-9def7a380328)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" podUID="92004b1b-d2d8-413a-be62-9def7a380328"
Sep 16 12:49:21.863586 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:21.863284    2579 scope.go:117] "RemoveContainer" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"
Sep 16 12:49:21.863586 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:21.863488    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7ljz6_openshift-console-operator(92004b1b-d2d8-413a-be62-9def7a380328)\"" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" podUID="92004b1b-d2d8-413a-be62-9def7a380328"
Sep 16 12:49:24.759338 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:24.759300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:24.759819 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:24.759368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:24.759819 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:24.759473    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:24.759819 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:24.759538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:32.75951845 +0000 UTC m=+78.798317568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:24.759819 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:24.759624    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:24.759819 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:24.759683    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls podName:cce6fb50-b915-40c7-a3d3-d934cbfaadc4 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:32.759669377 +0000 UTC m=+78.798468500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rcht6" (UID: "cce6fb50-b915-40c7-a3d3-d934cbfaadc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:25.876226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:25.876195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p2tq4" event={"ID":"3a0399ab-d7ad-410e-9383-8e8eaaeb106e","Type":"ContainerStarted","Data":"5012020690f0241af8de24176fe7eebf7a745876198ff9de5843e95b988e8ea6"}
Sep 16 12:49:25.876676 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:25.876308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:49:25.877698 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:25.877671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-74lv5" event={"ID":"d4820864-53d4-4b54-82dd-996b2be796ec","Type":"ContainerStarted","Data":"3364f15a94d44f8e49971f39bfeea9597933450326a4b25473c8b6222636ea9c"}
Sep 16 12:49:25.908260 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:25.908199    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p2tq4" podStartSLOduration=66.751770594 podStartE2EDuration="1m11.908177733s" totalImagesPullingTime="5.156407139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:25.90643323 +0000 UTC m=+71.945232359" watchObservedRunningTime="2026-09-16 12:49:25.908177733 +0000 UTC m=+71.946976858"
Sep 16 12:49:25.938727 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:25.938673    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-74lv5" podStartSLOduration=66.754193499 podStartE2EDuration="1m11.938655687s" totalImagesPullingTime="5.184462188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:25.938067912 +0000 UTC m=+71.976867036" watchObservedRunningTime="2026-09-16 12:49:25.938655687 +0000 UTC m=+71.977454809"
Sep 16 12:49:26.979949 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:26.979906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:26.979949 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:26.979945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:26.980434 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:49:26.980203    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle podName:c671399e-37d1-4307-8c61-a9464e26f327 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:42.980185337 +0000 UTC m=+89.018984453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle") pod "router-default-55fccc44b8-9v5dg" (UID: "c671399e-37d1-4307-8c61-a9464e26f327") : configmap references non-existent config key: service-ca.crt
Sep 16 12:49:26.982348 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:26.982326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c671399e-37d1-4307-8c61-a9464e26f327-metrics-certs\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:32.825343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:32.825300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:32.825343 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:32.825352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:32.825881 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:32.825838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:32.827601 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:32.827580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce6fb50-b915-40c7-a3d3-d934cbfaadc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rcht6\" (UID: \"cce6fb50-b915-40c7-a3d3-d934cbfaadc4\") " pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:32.886576 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:32.886539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rcht6"
Sep 16 12:49:33.159852 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:33.159821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rcht6"]
Sep 16 12:49:33.163918 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:33.163877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce6fb50_b915_40c7_a3d3_d934cbfaadc4.slice/crio-362ef5767eddae178fb7ed604380c4bbee2ff4e51b0624c7859f2e1cd4f51ece WatchSource:0}: Error finding container 362ef5767eddae178fb7ed604380c4bbee2ff4e51b0624c7859f2e1cd4f51ece: Status 404 returned error can't find the container with id 362ef5767eddae178fb7ed604380c4bbee2ff4e51b0624c7859f2e1cd4f51ece
Sep 16 12:49:33.904349 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:33.904315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rcht6" event={"ID":"cce6fb50-b915-40c7-a3d3-d934cbfaadc4","Type":"ContainerStarted","Data":"dc29afbdb12d6c57b3c8b3024cf1ea24ac5cd23630551c18961c663eba16a216"}
Sep 16 12:49:33.904652 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:33.904388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rcht6" event={"ID":"cce6fb50-b915-40c7-a3d3-d934cbfaadc4","Type":"ContainerStarted","Data":"362ef5767eddae178fb7ed604380c4bbee2ff4e51b0624c7859f2e1cd4f51ece"}
Sep 16 12:49:34.454522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.454489    2579 scope.go:117] "RemoveContainer" containerID="cf94b2eb061b1f1eeff4e6a13f683c985dc52d99a1b8001f4ac3360fabe340cc"
Sep 16 12:49:34.911632 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.911600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rcht6" event={"ID":"cce6fb50-b915-40c7-a3d3-d934cbfaadc4","Type":"ContainerStarted","Data":"af9f1406410f87b3fcc344543b6958f480b81219f1ca6e98ce91c12fdb1b66c0"}
Sep 16 12:49:34.914070 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.914032    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:49:34.914195 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.914108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" event={"ID":"92004b1b-d2d8-413a-be62-9def7a380328","Type":"ContainerStarted","Data":"5fff09c82743de12bce51739f498e1d555f20b9803731a06d4ff3ce3208ba9b9"}
Sep 16 12:49:34.914460 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.914412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:34.920871 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.920845    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6"
Sep 16 12:49:34.934929 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:34.934836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-7ljz6" podStartSLOduration=23.133863335 podStartE2EDuration="24.934816465s" totalImagesPullingTime="1.80095313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:34.934593705 +0000 UTC m=+80.973392828" watchObservedRunningTime="2026-09-16 12:49:34.934816465 +0000 UTC m=+80.973615589"
Sep 16 12:49:35.919228 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:35.919187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rcht6" event={"ID":"cce6fb50-b915-40c7-a3d3-d934cbfaadc4","Type":"ContainerStarted","Data":"799424c5475d7e0fc0bf169144418291325e09709cc0fe8e852ffaeb97c8a7eb"}
Sep 16 12:49:35.946043 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:35.945992    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rcht6" podStartSLOduration=18.074585277 podStartE2EDuration="19.945974593s" totalImagesPullingTime="1.871389316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:35.944618524 +0000 UTC m=+81.983417647" watchObservedRunningTime="2026-09-16 12:49:35.945974593 +0000 UTC m=+81.984773715"
Sep 16 12:49:42.124721 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.124610    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"]
Sep 16 12:49:42.127848 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.127822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.130870 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.130852    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:49:42.130974 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.130879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:49:42.132462 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.132446    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:49:42.132534 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.132515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:49:42.143839 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.143818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"]
Sep 16 12:49:42.198757 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.198724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d9b08de-fe6e-4819-b1ce-3e7623195e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.198916 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.198769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shvl\" (UniqueName: \"kubernetes.io/projected/5d9b08de-fe6e-4819-b1ce-3e7623195e40-kube-api-access-5shvl\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.198916 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.198853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9b08de-fe6e-4819-b1ce-3e7623195e40-tmp\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.225123 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.225093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68fd47758c-jwdmd"]
Sep 16 12:49:42.227899 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.227882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.230611 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.230589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:49:42.230904 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.230888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:49:42.231051 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.230926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:49:42.231299 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.231284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-27ln7\""
Sep 16 12:49:42.236085 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.236066    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:49:42.241736 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.241713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68fd47758c-jwdmd"]
Sep 16 12:49:42.299800 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d9b08de-fe6e-4819-b1ce-3e7623195e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.299984 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-installation-pull-secrets\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.299984 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shvl\" (UniqueName: \"kubernetes.io/projected/5d9b08de-fe6e-4819-b1ce-3e7623195e40-kube-api-access-5shvl\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.299984 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-certificates\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.299984 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-image-registry-private-configuration\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.299994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18890be2-3926-46fd-bb8b-43e68f22b2ca-ca-trust-extracted\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-trusted-ca\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9b08de-fe6e-4819-b1ce-3e7623195e40-tmp\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-tls\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-bound-sa-token\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh687\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-kube-api-access-nh687\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.300537 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.300435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9b08de-fe6e-4819-b1ce-3e7623195e40-tmp\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.302349 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.302334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d9b08de-fe6e-4819-b1ce-3e7623195e40-klusterlet-config\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.308961 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.308939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shvl\" (UniqueName: \"kubernetes.io/projected/5d9b08de-fe6e-4819-b1ce-3e7623195e40-kube-api-access-5shvl\") pod \"klusterlet-addon-workmgr-7897785798-kl5bz\" (UID: \"5d9b08de-fe6e-4819-b1ce-3e7623195e40\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.401141 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-installation-pull-secrets\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401297 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-certificates\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401297 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-image-registry-private-configuration\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401297 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18890be2-3926-46fd-bb8b-43e68f22b2ca-ca-trust-extracted\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401297 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-trusted-ca\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401498 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-tls\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401498 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-bound-sa-token\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401498 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh687\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-kube-api-access-nh687\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.401690 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.401604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18890be2-3926-46fd-bb8b-43e68f22b2ca-ca-trust-extracted\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.402347 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.402324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-trusted-ca\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.402456 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.402433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-certificates\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.403919 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.403892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-image-registry-private-configuration\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.404172 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.404156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-registry-tls\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.404172 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.404166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18890be2-3926-46fd-bb8b-43e68f22b2ca-installation-pull-secrets\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.411285 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.411267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh687\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-kube-api-access-nh687\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.412448 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.412427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18890be2-3926-46fd-bb8b-43e68f22b2ca-bound-sa-token\") pod \"image-registry-68fd47758c-jwdmd\" (UID: \"18890be2-3926-46fd-bb8b-43e68f22b2ca\") " pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.436883 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.436848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:42.537465 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.537429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.709595 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.709489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"]
Sep 16 12:49:42.712548 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:42.712523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b08de_fe6e_4819_b1ce_3e7623195e40.slice/crio-101e4aca113da727a1d79ff88e17991e5b1f90921df3bcd80e6f57bd311abf5a WatchSource:0}: Error finding container 101e4aca113da727a1d79ff88e17991e5b1f90921df3bcd80e6f57bd311abf5a: Status 404 returned error can't find the container with id 101e4aca113da727a1d79ff88e17991e5b1f90921df3bcd80e6f57bd311abf5a
Sep 16 12:49:42.811195 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.811166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68fd47758c-jwdmd"]
Sep 16 12:49:42.814336 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:42.814313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18890be2_3926_46fd_bb8b_43e68f22b2ca.slice/crio-4e593ba6afb458114ca1d1df78a0dfe0a501e48d2653e0839f413912bd7ea405 WatchSource:0}: Error finding container 4e593ba6afb458114ca1d1df78a0dfe0a501e48d2653e0839f413912bd7ea405: Status 404 returned error can't find the container with id 4e593ba6afb458114ca1d1df78a0dfe0a501e48d2653e0839f413912bd7ea405
Sep 16 12:49:42.942382 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.942341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd" event={"ID":"18890be2-3926-46fd-bb8b-43e68f22b2ca","Type":"ContainerStarted","Data":"3fa750f0bf25337a7b82ad9abc3a4ae3d44232db75ea5c8e91826291a94f9a4d"}
Sep 16 12:49:42.942583 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.942508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:49:42.942583 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.942548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd" event={"ID":"18890be2-3926-46fd-bb8b-43e68f22b2ca","Type":"ContainerStarted","Data":"4e593ba6afb458114ca1d1df78a0dfe0a501e48d2653e0839f413912bd7ea405"}
Sep 16 12:49:42.943579 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.943537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz" event={"ID":"5d9b08de-fe6e-4819-b1ce-3e7623195e40","Type":"ContainerStarted","Data":"101e4aca113da727a1d79ff88e17991e5b1f90921df3bcd80e6f57bd311abf5a"}
Sep 16 12:49:42.964803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:42.964707    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd" podStartSLOduration=0.96469155 podStartE2EDuration="964.69155ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:42.96345893 +0000 UTC m=+89.002258090" watchObservedRunningTime="2026-09-16 12:49:42.96469155 +0000 UTC m=+89.003490672"
Sep 16 12:49:43.007417 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.007377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:43.008272 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.008247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c671399e-37d1-4307-8c61-a9464e26f327-service-ca-bundle\") pod \"router-default-55fccc44b8-9v5dg\" (UID: \"c671399e-37d1-4307-8c61-a9464e26f327\") " pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:43.223655 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.223578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-chr7n\""
Sep 16 12:49:43.231267 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.231250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:43.503701 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.503549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55fccc44b8-9v5dg"]
Sep 16 12:49:43.507060 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:43.507022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc671399e_37d1_4307_8c61_a9464e26f327.slice/crio-57a52583d10160b6807568b41300eae4f66b598029d1f66f1e8a7333e3a21de9 WatchSource:0}: Error finding container 57a52583d10160b6807568b41300eae4f66b598029d1f66f1e8a7333e3a21de9: Status 404 returned error can't find the container with id 57a52583d10160b6807568b41300eae4f66b598029d1f66f1e8a7333e3a21de9
Sep 16 12:49:43.949555 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.949495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55fccc44b8-9v5dg" event={"ID":"c671399e-37d1-4307-8c61-a9464e26f327","Type":"ContainerStarted","Data":"21c457540016bf311612af943f9b3781610fddca03d0d837b585970b6fe2bdfd"}
Sep 16 12:49:43.949773 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.949635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55fccc44b8-9v5dg" event={"ID":"c671399e-37d1-4307-8c61-a9464e26f327","Type":"ContainerStarted","Data":"57a52583d10160b6807568b41300eae4f66b598029d1f66f1e8a7333e3a21de9"}
Sep 16 12:49:43.974037 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:43.972896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55fccc44b8-9v5dg" podStartSLOduration=32.972876024 podStartE2EDuration="32.972876024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:43.97217378 +0000 UTC m=+90.010972928" watchObservedRunningTime="2026-09-16 12:49:43.972876024 +0000 UTC m=+90.011675149"
Sep 16 12:49:44.232518 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:44.232439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:44.235854 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:44.235831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:44.953108 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:44.953080    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:44.954471 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:44.954449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55fccc44b8-9v5dg"
Sep 16 12:49:45.197648 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.197620    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:49:45.200736 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.200711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.206244 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c7h5m\""
Sep 16 12:49:45.206576 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:49:45.206695 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:49:45.206803 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:49:45.206962 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:49:45.207038 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.206967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:49:45.207099 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.207071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:49:45.211897 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.211879    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:49:45.221207 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.221184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:49:45.327156 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786bm\" (UniqueName: \"kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.327633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.327633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.327633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.327633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.327633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.327441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428064 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428243 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428243 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428243 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428399 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786bm\" (UniqueName: \"kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428399 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.428919 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.428893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.429117 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.429090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.429236 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.429126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.430907 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.430882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.431118 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.431100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.437739 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.437715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-786bm\" (UniqueName: \"kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm\") pod \"console-5db59c7879-m4m9n\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") " pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:45.511247 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:45.511158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:46.565282 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:46.565024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:49:46.567521 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:46.567494    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c3d4f0_ae56_4b03_af09_5fde55bb5dc4.slice/crio-9565100c6b441319e40d030efa87518c90b10bd10a62f1009817ad0951d1ca55 WatchSource:0}: Error finding container 9565100c6b441319e40d030efa87518c90b10bd10a62f1009817ad0951d1ca55: Status 404 returned error can't find the container with id 9565100c6b441319e40d030efa87518c90b10bd10a62f1009817ad0951d1ca55
Sep 16 12:49:46.965600 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:46.965541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz" event={"ID":"5d9b08de-fe6e-4819-b1ce-3e7623195e40","Type":"ContainerStarted","Data":"7538b6133bdc4572489f9421a64c032aaa99cf03dfe897d5f525dc7b22a9f160"}
Sep 16 12:49:46.965836 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:46.965821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:46.967081 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:46.967048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db59c7879-m4m9n" event={"ID":"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4","Type":"ContainerStarted","Data":"9565100c6b441319e40d030efa87518c90b10bd10a62f1009817ad0951d1ca55"}
Sep 16 12:49:46.967577 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:46.967548    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz"
Sep 16 12:49:47.009537 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:47.009484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7897785798-kl5bz" podStartSLOduration=1.349658514 podStartE2EDuration="5.009466785s" totalImagesPullingTime="3.659808271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:46.987870781 +0000 UTC m=+93.026669903" watchObservedRunningTime="2026-09-16 12:49:47.009466785 +0000 UTC m=+93.048265907"
Sep 16 12:49:49.981735 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:49.981696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db59c7879-m4m9n" event={"ID":"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4","Type":"ContainerStarted","Data":"0943d5dbed26310a4ca666eb2addb14c62108d11d92956d03ca285364dc9b8b5"}
Sep 16 12:49:50.001550 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.001504    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db59c7879-m4m9n" podStartSLOduration=2.465982052 podStartE2EDuration="5.001488423s" totalImagesPullingTime="2.535506371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:50.000107493 +0000 UTC m=+96.038906648" watchObservedRunningTime="2026-09-16 12:49:50.001488423 +0000 UTC m=+96.040287546"
Sep 16 12:49:50.152664 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.152633    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"]
Sep 16 12:49:50.155740 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.155726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.158866    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rwdfq\""
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.159279    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.159371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.159437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.159604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:49:50.161426 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.159604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:49:50.166381 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.166353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"]
Sep 16 12:49:50.268325 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.268232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.268325 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.268284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2b254-98d3-42a4-a47c-780941d2b3d2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.268586 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.268349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc4s\" (UniqueName: \"kubernetes.io/projected/c9f2b254-98d3-42a4-a47c-780941d2b3d2-kube-api-access-9cc4s\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.268586 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.268407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.369317 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.369285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.369491 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.369349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.369491 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.369380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2b254-98d3-42a4-a47c-780941d2b3d2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.369491 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.369402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc4s\" (UniqueName: \"kubernetes.io/projected/c9f2b254-98d3-42a4-a47c-780941d2b3d2-kube-api-access-9cc4s\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.370149 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.370125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2b254-98d3-42a4-a47c-780941d2b3d2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.371748 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.371722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.371831 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.371797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9f2b254-98d3-42a4-a47c-780941d2b3d2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.378574 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.378533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc4s\" (UniqueName: \"kubernetes.io/projected/c9f2b254-98d3-42a4-a47c-780941d2b3d2-kube-api-access-9cc4s\") pod \"prometheus-operator-68bcd7f987-v5bmn\" (UID: \"c9f2b254-98d3-42a4-a47c-780941d2b3d2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.468299 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.468258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"
Sep 16 12:49:50.740482 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.740323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn"]
Sep 16 12:49:50.742921 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:50.742893    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f2b254_98d3_42a4_a47c_780941d2b3d2.slice/crio-3d8657371cad4eab53319d718011f3528e7f3cf6da6fcfa05b14a9e72f0b4c88 WatchSource:0}: Error finding container 3d8657371cad4eab53319d718011f3528e7f3cf6da6fcfa05b14a9e72f0b4c88: Status 404 returned error can't find the container with id 3d8657371cad4eab53319d718011f3528e7f3cf6da6fcfa05b14a9e72f0b4c88
Sep 16 12:49:50.985933 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:50.985900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn" event={"ID":"c9f2b254-98d3-42a4-a47c-780941d2b3d2","Type":"ContainerStarted","Data":"3d8657371cad4eab53319d718011f3528e7f3cf6da6fcfa05b14a9e72f0b4c88"}
Sep 16 12:49:51.277193 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.277108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:51.279795 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.279761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c3bcc27-8289-4ce0-896f-f530c2a30fba-metrics-tls\") pod \"dns-default-cm276\" (UID: \"0c3bcc27-8289-4ce0-896f-f530c2a30fba\") " pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:51.290552 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.290528    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwp6v\""
Sep 16 12:49:51.297898 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.297866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:51.592332 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.592254    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cm276"]
Sep 16 12:49:51.595303 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:51.595264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3bcc27_8289_4ce0_896f_f530c2a30fba.slice/crio-bf16fa2590e99ce108a07cb88731d99df8b18260f407d669bfba4ad52e5a25fc WatchSource:0}: Error finding container bf16fa2590e99ce108a07cb88731d99df8b18260f407d669bfba4ad52e5a25fc: Status 404 returned error can't find the container with id bf16fa2590e99ce108a07cb88731d99df8b18260f407d669bfba4ad52e5a25fc
Sep 16 12:49:51.990555 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:51.990520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cm276" event={"ID":"0c3bcc27-8289-4ce0-896f-f530c2a30fba","Type":"ContainerStarted","Data":"bf16fa2590e99ce108a07cb88731d99df8b18260f407d669bfba4ad52e5a25fc"}
Sep 16 12:49:52.997542 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:52.997436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn" event={"ID":"c9f2b254-98d3-42a4-a47c-780941d2b3d2","Type":"ContainerStarted","Data":"51ef06d70c1e4b09fd976a974ea8f265efb34534786bdf5da0bea63eba2613e8"}
Sep 16 12:49:52.997542 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:52.997508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn" event={"ID":"c9f2b254-98d3-42a4-a47c-780941d2b3d2","Type":"ContainerStarted","Data":"6c5d6cd6117c15488e8feaa8bceecb4901cb38791e3dab9976780d7e30d0bae0"}
Sep 16 12:49:53.017222 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:53.017169    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-v5bmn" podStartSLOduration=1.7925033780000001 podStartE2EDuration="3.017147489s" totalImagesPullingTime="1.224644111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:53.014920216 +0000 UTC m=+99.053719335" watchObservedRunningTime="2026-09-16 12:49:53.017147489 +0000 UTC m=+99.055946614"
Sep 16 12:49:54.003039 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.003003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cm276" event={"ID":"0c3bcc27-8289-4ce0-896f-f530c2a30fba","Type":"ContainerStarted","Data":"6ef78be321e4a515c425fec3350bff898af3d2d3498a521813ebfc301f4e6997"}
Sep 16 12:49:54.003393 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.003059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cm276" event={"ID":"0c3bcc27-8289-4ce0-896f-f530c2a30fba","Type":"ContainerStarted","Data":"c533ff75139a3d6d4eec972257f29d8616048dd8fd7b12d736e1deba88c6acda"}
Sep 16 12:49:54.023992 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.023933    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cm276" podStartSLOduration=65.517954679 podStartE2EDuration="1m7.023918542s" totalImagesPullingTime="1.505963863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:54.022018827 +0000 UTC m=+100.060817952" watchObservedRunningTime="2026-09-16 12:49:54.023918542 +0000 UTC m=+100.062717665"
Sep 16 12:49:54.581819 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.581785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-df7jd"]
Sep 16 12:49:54.585777 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.585755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.589849 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.589818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:49:54.590108 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.590091    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ksdcc\""
Sep 16 12:49:54.590283 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.590268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:49:54.590526 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.590504    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:49:54.709743 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-metrics-client-ca\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.709935 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-tls\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.709935 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.709935 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-textfile\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.710092 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-sys\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.710092 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.709990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g88kz\" (UniqueName: \"kubernetes.io/projected/52efc589-4291-480d-8014-961d6ec84d43-kube-api-access-g88kz\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.710092 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.710047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-root\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.710226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.710093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-wtmp\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.710226 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.710112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-accelerators-collector-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811287 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-sys\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811464 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g88kz\" (UniqueName: \"kubernetes.io/projected/52efc589-4291-480d-8014-961d6ec84d43-kube-api-access-g88kz\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811464 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-sys\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811464 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-root\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811464 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-wtmp\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-accelerators-collector-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-root\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-metrics-client-ca\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-tls\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-wtmp\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811704 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.811984 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.811704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-textfile\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.812184 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.812164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-textfile\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.812327 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.812173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-accelerators-collector-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.812457 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.812350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52efc589-4291-480d-8014-961d6ec84d43-metrics-client-ca\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.814006 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.813985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.814807 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.814789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/52efc589-4291-480d-8014-961d6ec84d43-node-exporter-tls\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.820737 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.820710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g88kz\" (UniqueName: \"kubernetes.io/projected/52efc589-4291-480d-8014-961d6ec84d43-kube-api-access-g88kz\") pod \"node-exporter-df7jd\" (UID: \"52efc589-4291-480d-8014-961d6ec84d43\") " pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.898347 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:54.898315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-df7jd"
Sep 16 12:49:54.909831 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:49:54.909800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52efc589_4291_480d_8014_961d6ec84d43.slice/crio-0956ed65ce62655f8b53227cf80ff296e9fbf5bb5840d96e46097c903938eb1e WatchSource:0}: Error finding container 0956ed65ce62655f8b53227cf80ff296e9fbf5bb5840d96e46097c903938eb1e: Status 404 returned error can't find the container with id 0956ed65ce62655f8b53227cf80ff296e9fbf5bb5840d96e46097c903938eb1e
Sep 16 12:49:55.007837 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:55.007800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df7jd" event={"ID":"52efc589-4291-480d-8014-961d6ec84d43","Type":"ContainerStarted","Data":"0956ed65ce62655f8b53227cf80ff296e9fbf5bb5840d96e46097c903938eb1e"}
Sep 16 12:49:55.008303 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:55.007941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cm276"
Sep 16 12:49:55.511513 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:55.511476    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:55.511718 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:55.511541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:55.517471 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:55.517447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:56.014603 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:56.014546    2579 generic.go:356] "Generic (PLEG): container finished" podID="52efc589-4291-480d-8014-961d6ec84d43" containerID="89f605631c267060044e3808bc1088cff106f5035ca566584df64b54a18721e4" exitCode=0
Sep 16 12:49:56.015034 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:56.014606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df7jd" event={"ID":"52efc589-4291-480d-8014-961d6ec84d43","Type":"ContainerDied","Data":"89f605631c267060044e3808bc1088cff106f5035ca566584df64b54a18721e4"}
Sep 16 12:49:56.019807 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:56.019777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:49:56.883907 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:56.883874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p2tq4"
Sep 16 12:49:57.020882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:57.020842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df7jd" event={"ID":"52efc589-4291-480d-8014-961d6ec84d43","Type":"ContainerStarted","Data":"e315c157e9d624d45cc3ae73a30cf0c39ae8e4a03a39a88c552970c3f73b2fc8"}
Sep 16 12:49:57.021324 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:57.020914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df7jd" event={"ID":"52efc589-4291-480d-8014-961d6ec84d43","Type":"ContainerStarted","Data":"06895c33ada3b8a0f4380ca2a7c4474d5e68fc5ec8783319296d8e0f32c84438"}
Sep 16 12:49:57.040402 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:49:57.040354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-df7jd" podStartSLOduration=2.0815044719999998 podStartE2EDuration="3.040339028s" totalImagesPullingTime="888.711642ms" totalInitContainerRuntime="70.122914ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:57.039730473 +0000 UTC m=+103.078529607" watchObservedRunningTime="2026-09-16 12:49:57.040339028 +0000 UTC m=+103.079138151"
Sep 16 12:50:03.953968 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:03.953937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68fd47758c-jwdmd"
Sep 16 12:50:05.017663 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:05.017628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cm276"
Sep 16 12:50:12.959376 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:12.959337    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:50:20.942184 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:20.942120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 12:50:20.951395 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:20.951368    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jldt8_a07394e1-5dc3-4faa-a8ec-517ab84483e2/serve-healthcheck-canary/0.log"
Sep 16 12:50:21.103878 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:21.103850    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b269419-28c4-4b80-80aa-50956087db1b" containerID="2ee791b7d451db88f3cd84af627c984b02a95f7547e1cf2c8952d4b96c0c0dd9" exitCode=0
Sep 16 12:50:21.104004 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:21.103889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55" event={"ID":"5b269419-28c4-4b80-80aa-50956087db1b","Type":"ContainerDied","Data":"2ee791b7d451db88f3cd84af627c984b02a95f7547e1cf2c8952d4b96c0c0dd9"}
Sep 16 12:50:21.104214 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:21.104200    2579 scope.go:117] "RemoveContainer" containerID="2ee791b7d451db88f3cd84af627c984b02a95f7547e1cf2c8952d4b96c0c0dd9"
Sep 16 12:50:22.110921 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:22.110887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-9lf55" event={"ID":"5b269419-28c4-4b80-80aa-50956087db1b","Type":"ContainerStarted","Data":"b0d4291a2f80a2413b5eefdd8885b73b28765299b1c7ce50b8a2666d658a63f2"}
Sep 16 12:50:23.250034 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:23.249999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:50:23.252280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:23.252260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5487dc67-8a7e-4ff9-8a18-85d8f102b01e-metrics-certs\") pod \"network-metrics-daemon-57zxt\" (UID: \"5487dc67-8a7e-4ff9-8a18-85d8f102b01e\") " pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:50:23.327254 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:23.327223    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v25dx\""
Sep 16 12:50:23.335386 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:23.335363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57zxt"
Sep 16 12:50:23.646314 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:23.646289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57zxt"]
Sep 16 12:50:23.648716 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:50:23.648692    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487dc67_8a7e_4ff9_8a18_85d8f102b01e.slice/crio-936c3ee96e84bc773421408daefc65ccd87ba2a29b9d593e5111f796a2129b2a WatchSource:0}: Error finding container 936c3ee96e84bc773421408daefc65ccd87ba2a29b9d593e5111f796a2129b2a: Status 404 returned error can't find the container with id 936c3ee96e84bc773421408daefc65ccd87ba2a29b9d593e5111f796a2129b2a
Sep 16 12:50:24.118616 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:24.118536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57zxt" event={"ID":"5487dc67-8a7e-4ff9-8a18-85d8f102b01e","Type":"ContainerStarted","Data":"936c3ee96e84bc773421408daefc65ccd87ba2a29b9d593e5111f796a2129b2a"}
Sep 16 12:50:25.123726 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:25.123695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57zxt" event={"ID":"5487dc67-8a7e-4ff9-8a18-85d8f102b01e","Type":"ContainerStarted","Data":"421e25f55d7e4a83cfc11769dc8ec4003c5611751d61f5bfe4c57564086471fc"}
Sep 16 12:50:26.132747 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:26.132709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57zxt" event={"ID":"5487dc67-8a7e-4ff9-8a18-85d8f102b01e","Type":"ContainerStarted","Data":"37753ef6c17f71e3f0c32fe87255c99b81d84db00dee3d1d1887900248e5cd6c"}
Sep 16 12:50:26.151816 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:26.151763    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-57zxt" podStartSLOduration=130.922705728 podStartE2EDuration="2m12.151748802s" totalImagesPullingTime="1.229043074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:26.150499963 +0000 UTC m=+132.189299085" watchObservedRunningTime="2026-09-16 12:50:26.151748802 +0000 UTC m=+132.190547965"
Sep 16 12:50:37.984054 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:37.983999    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5db59c7879-m4m9n" podUID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" containerName="console" containerID="cri-o://0943d5dbed26310a4ca666eb2addb14c62108d11d92956d03ca285364dc9b8b5" gracePeriod=15
Sep 16 12:50:38.177300 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.177256    2579 generic.go:356] "Generic (PLEG): container finished" podID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" containerID="0943d5dbed26310a4ca666eb2addb14c62108d11d92956d03ca285364dc9b8b5" exitCode=0
Sep 16 12:50:38.177437 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.177329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db59c7879-m4m9n" event={"ID":"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4","Type":"ContainerDied","Data":"0943d5dbed26310a4ca666eb2addb14c62108d11d92956d03ca285364dc9b8b5"}
Sep 16 12:50:38.214903 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.214881    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:50:38.266225 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266392 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266224    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-786bm\" (UniqueName: \"kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266392 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266317    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266392 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266536 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266536 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca\") pod \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\" (UID: \"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4\") "
Sep 16 12:50:38.266679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:38.266787 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config" (OuterVolumeSpecName: "console-config") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:38.266916 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266889    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca" (OuterVolumeSpecName: "service-ca") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:38.266973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266903    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-config\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:38.266973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.266944    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-oauth-serving-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:38.268587 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.268547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:38.268685 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.268608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:38.268685 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.268612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm" (OuterVolumeSpecName: "kube-api-access-786bm") pod "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" (UID: "26c3d4f0-ae56-4b03-af09-5fde55bb5dc4"). InnerVolumeSpecName "kube-api-access-786bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:38.367882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.367842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786bm\" (UniqueName: \"kubernetes.io/projected/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-kube-api-access-786bm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:38.367882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.367872    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-serving-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:38.367882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.367881    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-console-oauth-config\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:38.367882 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:38.367890    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4-service-ca\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:50:39.182232 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:39.182193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db59c7879-m4m9n" event={"ID":"26c3d4f0-ae56-4b03-af09-5fde55bb5dc4","Type":"ContainerDied","Data":"9565100c6b441319e40d030efa87518c90b10bd10a62f1009817ad0951d1ca55"}
Sep 16 12:50:39.182232 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:39.182224    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db59c7879-m4m9n"
Sep 16 12:50:39.182789 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:39.182302    2579 scope.go:117] "RemoveContainer" containerID="0943d5dbed26310a4ca666eb2addb14c62108d11d92956d03ca285364dc9b8b5"
Sep 16 12:50:39.200510 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:39.200486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:50:39.206765 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:39.206745    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db59c7879-m4m9n"]
Sep 16 12:50:40.457044 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:50:40.457013    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" path="/var/lib/kubelet/pods/26c3d4f0-ae56-4b03-af09-5fde55bb5dc4/volumes"
Sep 16 12:51:33.377801 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.377768    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:51:33.378264 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.378030    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" containerName="console"
Sep 16 12:51:33.378264 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.378040    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" containerName="console"
Sep 16 12:51:33.378264 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.378101    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c3d4f0-ae56-4b03-af09-5fde55bb5dc4" containerName="console"
Sep 16 12:51:33.382116 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.382093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.388420 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.388378    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:51:33.388602 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.388432    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:51:33.389833 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.389815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:51:33.392217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.392081    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:51:33.392217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.392117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c7h5m\""
Sep 16 12:51:33.392217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.392149    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:51:33.392217 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.392122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:51:33.396430 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.396405    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:51:33.396767 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.396746    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:51:33.411268 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.411238    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:51:33.486635 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7j4\" (UniqueName: \"kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.486821 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.486821 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.486821 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.486821 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.487002 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.487002 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.486884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587706 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587706 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587959 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7j4\" (UniqueName: \"kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587959 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587959 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587959 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.587959 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.587829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.588494 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.588471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.588634 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.588611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.588701 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.588611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.588836 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.588812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.590415 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.590390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.590496 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.590472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.596669 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.596649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7j4\" (UniqueName: \"kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4\") pod \"console-6884f87585-5vcsc\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") " pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.691808 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.691771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:33.965323 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:33.965296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:51:33.967753 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:51:33.967726    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a39a67_471f_4fd2_83ae_c1778c1a119f.slice/crio-c3aed4dd2d77b4aa66f4ddd9572867a4bddb574e1b3f129075d24a0e1b4c35c2 WatchSource:0}: Error finding container c3aed4dd2d77b4aa66f4ddd9572867a4bddb574e1b3f129075d24a0e1b4c35c2: Status 404 returned error can't find the container with id c3aed4dd2d77b4aa66f4ddd9572867a4bddb574e1b3f129075d24a0e1b4c35c2
Sep 16 12:51:34.370347 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:34.370250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6884f87585-5vcsc" event={"ID":"b1a39a67-471f-4fd2-83ae-c1778c1a119f","Type":"ContainerStarted","Data":"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"}
Sep 16 12:51:34.370347 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:34.370312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6884f87585-5vcsc" event={"ID":"b1a39a67-471f-4fd2-83ae-c1778c1a119f","Type":"ContainerStarted","Data":"c3aed4dd2d77b4aa66f4ddd9572867a4bddb574e1b3f129075d24a0e1b4c35c2"}
Sep 16 12:51:34.389093 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:34.389038    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6884f87585-5vcsc" podStartSLOduration=1.389018277 podStartE2EDuration="1.389018277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:34.388636347 +0000 UTC m=+200.427435522" watchObservedRunningTime="2026-09-16 12:51:34.389018277 +0000 UTC m=+200.427817403"
Sep 16 12:51:43.692317 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:43.692276    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:43.692934 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:43.692354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:43.696900 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:43.696880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:51:44.412659 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:51:44.412631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:53:14.327518 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:53:14.327485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:53:14.331468 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:53:14.331443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:53:14.336570 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:53:14.336534    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:55:09.608913 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.608874    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"]
Sep 16 12:55:09.612174 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.612145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.615613 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.615589    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 12:55:09.615723 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.615707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:55:09.616048 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.616033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pbp8d\""
Sep 16 12:55:09.628046 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.628003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"]
Sep 16 12:55:09.731114 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.731079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.731114 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.731116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2gn\" (UniqueName: \"kubernetes.io/projected/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-kube-api-access-4j2gn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.831714 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.831683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.831814 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.831725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2gn\" (UniqueName: \"kubernetes.io/projected/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-kube-api-access-4j2gn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.832068 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.832049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.840853 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.840830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2gn\" (UniqueName: \"kubernetes.io/projected/b4f790b2-f9a2-4ebc-a13c-a534232ee47f-kube-api-access-4j2gn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hrhv9\" (UID: \"b4f790b2-f9a2-4ebc-a13c-a534232ee47f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:09.920781 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:09.920752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"
Sep 16 12:55:10.221293 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:10.221219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9"]
Sep 16 12:55:10.224426 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:55:10.224401    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f790b2_f9a2_4ebc_a13c_a534232ee47f.slice/crio-608017013f279df0afc32544232d961dc4e23e226cbef681e9ed0f7a299776e2 WatchSource:0}: Error finding container 608017013f279df0afc32544232d961dc4e23e226cbef681e9ed0f7a299776e2: Status 404 returned error can't find the container with id 608017013f279df0afc32544232d961dc4e23e226cbef681e9ed0f7a299776e2
Sep 16 12:55:10.227013 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:10.226998    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:55:11.124388 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:11.124352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9" event={"ID":"b4f790b2-f9a2-4ebc-a13c-a534232ee47f","Type":"ContainerStarted","Data":"608017013f279df0afc32544232d961dc4e23e226cbef681e9ed0f7a299776e2"}
Sep 16 12:55:13.136464 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:13.136373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9" event={"ID":"b4f790b2-f9a2-4ebc-a13c-a534232ee47f","Type":"ContainerStarted","Data":"5e88ec80664b1a457f91ec48d410d600f009f62c116c17e9731dec246b4b813d"}
Sep 16 12:55:15.022840 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.022785    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hrhv9" podStartSLOduration=3.390123824 podStartE2EDuration="6.022767573s" totalImagesPullingTime="2.632643749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:13.165585422 +0000 UTC m=+419.204384538" watchObservedRunningTime="2026-09-16 12:55:15.022767573 +0000 UTC m=+421.061566696"
Sep 16 12:55:15.023594 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.023546    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7tl7"]
Sep 16 12:55:15.026767 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.026747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.031462 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.031441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:55:15.031795 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.031757    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:55:15.031795 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.031771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j9cz5\""
Sep 16 12:55:15.048589 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.048547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7tl7"]
Sep 16 12:55:15.074311 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.074276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2nv\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-kube-api-access-rx2nv\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.074482 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.074341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.175647 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.175613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.175821 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.175673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2nv\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-kube-api-access-rx2nv\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.191245 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.191211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2nv\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-kube-api-access-rx2nv\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.194665 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.194643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p7tl7\" (UID: \"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.345448 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.345368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:15.651396 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:15.651371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p7tl7"]
Sep 16 12:55:15.652929 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:55:15.652907    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99aa6b48_12f8_4eb5_8d37_9a9cb49acf1d.slice/crio-3983cf29c18ff0e8305fc6ca8011ff178668fb9f38bdc32d1c8caaa5a202bb77 WatchSource:0}: Error finding container 3983cf29c18ff0e8305fc6ca8011ff178668fb9f38bdc32d1c8caaa5a202bb77: Status 404 returned error can't find the container with id 3983cf29c18ff0e8305fc6ca8011ff178668fb9f38bdc32d1c8caaa5a202bb77
Sep 16 12:55:16.148159 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:16.148124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7" event={"ID":"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d","Type":"ContainerStarted","Data":"3983cf29c18ff0e8305fc6ca8011ff178668fb9f38bdc32d1c8caaa5a202bb77"}
Sep 16 12:55:19.161903 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:19.161870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7" event={"ID":"99aa6b48-12f8-4eb5-8d37-9a9cb49acf1d","Type":"ContainerStarted","Data":"4c13b154a4dfc044b7b5757b649ddc99418ab763f63f21bcbc9009b877296c81"}
Sep 16 12:55:19.162285 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:19.161972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:24.532908 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.532845    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7" podStartSLOduration=7.9730555 podStartE2EDuration="10.532827863s" totalImagesPullingTime="2.559772363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:19.186109665 +0000 UTC m=+425.224908787" watchObservedRunningTime="2026-09-16 12:55:24.532827863 +0000 UTC m=+430.571626985"
Sep 16 12:55:24.533531 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.533516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pc4jr"]
Sep 16 12:55:24.536744 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.536728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.539633 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.539613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x897q\""
Sep 16 12:55:24.557975 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.557936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pc4jr"]
Sep 16 12:55:24.649805 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.649769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xzm\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-kube-api-access-z6xzm\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.649805 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.649807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.750582 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.750522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xzm\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-kube-api-access-z6xzm\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.750582 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.750586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.773491 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.773456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-bound-sa-token\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.773642 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.773584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xzm\" (UniqueName: \"kubernetes.io/projected/ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b-kube-api-access-z6xzm\") pod \"cert-manager-55b4d954d5-pc4jr\" (UID: \"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b\") " pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:24.845661 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:24.845546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pc4jr"
Sep 16 12:55:25.123477 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:25.123407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pc4jr"]
Sep 16 12:55:25.125751 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:55:25.125719    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca3c36fc_2cc7_4b4c_bdf5_27afa752fa2b.slice/crio-c91f56e3161316dec71d1d4d5dec3a6e57d9c34fdfe4d6dca0b0727f8be5ca0f WatchSource:0}: Error finding container c91f56e3161316dec71d1d4d5dec3a6e57d9c34fdfe4d6dca0b0727f8be5ca0f: Status 404 returned error can't find the container with id c91f56e3161316dec71d1d4d5dec3a6e57d9c34fdfe4d6dca0b0727f8be5ca0f
Sep 16 12:55:25.168329 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:25.168306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p7tl7"
Sep 16 12:55:25.186106 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:25.186083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pc4jr" event={"ID":"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b","Type":"ContainerStarted","Data":"c91f56e3161316dec71d1d4d5dec3a6e57d9c34fdfe4d6dca0b0727f8be5ca0f"}
Sep 16 12:55:26.191104 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:26.191071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pc4jr" event={"ID":"ca3c36fc-2cc7-4b4c-bdf5-27afa752fa2b","Type":"ContainerStarted","Data":"842dbda6035a4bcaf603cd531dedd6bed4c58f6a299273678873129a98daadfe"}
Sep 16 12:55:26.209680 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:26.209629    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pc4jr" podStartSLOduration=2.209612702 podStartE2EDuration="2.209612702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:26.20781845 +0000 UTC m=+432.246617575" watchObservedRunningTime="2026-09-16 12:55:26.209612702 +0000 UTC m=+432.248411824"
Sep 16 12:55:29.171899 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:29.171851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:55:53.736822 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.736742    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"]
Sep 16 12:55:53.739156 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.739135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.744372 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.744346    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:55:53.744509 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.744435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-p6hvt\""
Sep 16 12:55:53.783980 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.783947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"]
Sep 16 12:55:53.888251 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v298q\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-kube-api-access-v298q\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888442 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888710 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.888710 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.888489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989522 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v298q\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-kube-api-access-v298q\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.989866 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.989736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.990166 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.990129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.990166 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.990162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.990341 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.990207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.990341 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.990228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.990341 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.990293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.991996 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.991972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.992101 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.992033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.999138 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.999112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:53.999328 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:53.999310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v298q\" (UniqueName: \"kubernetes.io/projected/2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75-kube-api-access-v298q\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq\" (UID: \"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:54.049357 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.049316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:54.191132 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.191067    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6884f87585-5vcsc" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerName="console" containerID="cri-o://c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5" gracePeriod=15
Sep 16 12:55:54.330075 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.330035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"]
Sep 16 12:55:54.332869 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:55:54.332840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2002c1fc_36a5_4cf5_bb6a_7c21fc1bbb75.slice/crio-51966f0453215e2f6559c22a8c19c3a15e3bd5db5ad581adbdd7f8ff891c76cd WatchSource:0}: Error finding container 51966f0453215e2f6559c22a8c19c3a15e3bd5db5ad581adbdd7f8ff891c76cd: Status 404 returned error can't find the container with id 51966f0453215e2f6559c22a8c19c3a15e3bd5db5ad581adbdd7f8ff891c76cd
Sep 16 12:55:54.422130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.422105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:55:54.496281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496530 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg7j4\" (UniqueName: \"kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496530 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496530 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496530 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496530 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca\") pod \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\" (UID: \"b1a39a67-471f-4fd2-83ae-c1778c1a119f\") "
Sep 16 12:55:54.496801 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:54.496962 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:54.496962 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.496949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config" (OuterVolumeSpecName: "console-config") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:54.497079 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.497012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:54.498610 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.498589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4" (OuterVolumeSpecName: "kube-api-access-qg7j4") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "kube-api-access-qg7j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:54.499019 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.499002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:54.499075 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.499017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1a39a67-471f-4fd2-83ae-c1778c1a119f" (UID: "b1a39a67-471f-4fd2-83ae-c1778c1a119f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597552    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-config\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597608    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-service-ca\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597617    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-serving-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597626    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-oauth-serving-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597635    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg7j4\" (UniqueName: \"kubernetes.io/projected/b1a39a67-471f-4fd2-83ae-c1778c1a119f-kube-api-access-qg7j4\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597643    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1a39a67-471f-4fd2-83ae-c1778c1a119f-console-oauth-config\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:54.597929 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:54.597664    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a39a67-471f-4fd2-83ae-c1778c1a119f-trusted-ca-bundle\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:55:55.305056 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.304973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" event={"ID":"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75","Type":"ContainerStarted","Data":"51966f0453215e2f6559c22a8c19c3a15e3bd5db5ad581adbdd7f8ff891c76cd"}
Sep 16 12:55:55.306843 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.306814    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerID="c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5" exitCode=0
Sep 16 12:55:55.306973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.306879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6884f87585-5vcsc" event={"ID":"b1a39a67-471f-4fd2-83ae-c1778c1a119f","Type":"ContainerDied","Data":"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"}
Sep 16 12:55:55.306973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.306882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6884f87585-5vcsc"
Sep 16 12:55:55.306973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.306920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6884f87585-5vcsc" event={"ID":"b1a39a67-471f-4fd2-83ae-c1778c1a119f","Type":"ContainerDied","Data":"c3aed4dd2d77b4aa66f4ddd9572867a4bddb574e1b3f129075d24a0e1b4c35c2"}
Sep 16 12:55:55.306973 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.306949    2579 scope.go:117] "RemoveContainer" containerID="c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"
Sep 16 12:55:55.335206 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.335168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:55:55.338528 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.338494    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6884f87585-5vcsc"]
Sep 16 12:55:55.389788 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.389767    2579 scope.go:117] "RemoveContainer" containerID="c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"
Sep 16 12:55:55.390214 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:55:55.390183    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5\": container with ID starting with c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5 not found: ID does not exist" containerID="c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"
Sep 16 12:55:55.390289 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.390228    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5"} err="failed to get container status \"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5\": rpc error: code = NotFound desc = could not find container \"c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5\": container with ID starting with c21d95258ef28899350e3dbfe5274d72846aa43e002584928d8d9cb68c549cb5 not found: ID does not exist"
Sep 16 12:55:55.410488 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.410449    2579 patch_prober.go:28] interesting pod/console-6884f87585-5vcsc container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.17:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:55:55.410678 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:55.410514    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6884f87585-5vcsc" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerName="console" probeResult="failure" output="Get \"https://10.134.0.17:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:55:56.457967 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:56.457934    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" path="/var/lib/kubelet/pods/b1a39a67-471f-4fd2-83ae-c1778c1a119f/volumes"
Sep 16 12:55:56.995936 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:56.995884    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:55:56.996058 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:56.995968    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:55:56.996058 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:56.995999    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:55:57.318646 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:57.318543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" event={"ID":"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75","Type":"ContainerStarted","Data":"bc511f150a007fd47ab5d1db32470894f0719570fd80e3d281a580e4ef71f8c7"}
Sep 16 12:55:57.345473 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:57.345415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podStartSLOduration=1.6849332160000001 podStartE2EDuration="4.345394964s" totalImagesPullingTime="2.660461748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:57.343848677 +0000 UTC m=+463.382647802" watchObservedRunningTime="2026-09-16 12:55:57.345394964 +0000 UTC m=+463.384194089"
Sep 16 12:55:58.050180 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:58.050138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:58.054910 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:58.054884    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:58.323505 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:58.323420    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:55:58.324679 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:55:58.324656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 12:56:00.243855 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.243818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"]
Sep 16 12:56:00.244260 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.244092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerName="console"
Sep 16 12:56:00.244260 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.244103    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerName="console"
Sep 16 12:56:00.244260 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.244160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a39a67-471f-4fd2-83ae-c1778c1a119f" containerName="console"
Sep 16 12:56:00.246841 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.246822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.251280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.251257    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:56:00.251751 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.251733    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:56:00.251818 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.251804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:56:00.252462 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.252439    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-c268j\""
Sep 16 12:56:00.252551 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.252489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:56:00.252551 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.252494    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:56:00.259804 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.259779    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"]
Sep 16 12:56:00.347185 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.347148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gvt\" (UniqueName: \"kubernetes.io/projected/e742f57c-4756-49fc-a306-2d69e9107678-kube-api-access-r5gvt\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.347185 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.347188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-metrics-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.347384 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.347260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.347384 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.347318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e742f57c-4756-49fc-a306-2d69e9107678-manager-config\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.448510 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.448479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gvt\" (UniqueName: \"kubernetes.io/projected/e742f57c-4756-49fc-a306-2d69e9107678-kube-api-access-r5gvt\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.448653 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.448520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-metrics-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.448653 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.448549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.448653 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.448628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e742f57c-4756-49fc-a306-2d69e9107678-manager-config\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.449386 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.449358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e742f57c-4756-49fc-a306-2d69e9107678-manager-config\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.451067 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.451043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-metrics-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.451168 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.451094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e742f57c-4756-49fc-a306-2d69e9107678-cert\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.462462 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.462434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gvt\" (UniqueName: \"kubernetes.io/projected/e742f57c-4756-49fc-a306-2d69e9107678-kube-api-access-r5gvt\") pod \"lws-controller-manager-6cc7b8d69b-h629m\" (UID: \"e742f57c-4756-49fc-a306-2d69e9107678\") " pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.555708 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.555602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:00.826355 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:00.826279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"]
Sep 16 12:56:00.829267 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:56:00.829240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode742f57c_4756_49fc_a306_2d69e9107678.slice/crio-da47666b55217a77a65b2f4b6ced983db6f3542d8e59ba2d40ebacec5f082591 WatchSource:0}: Error finding container da47666b55217a77a65b2f4b6ced983db6f3542d8e59ba2d40ebacec5f082591: Status 404 returned error can't find the container with id da47666b55217a77a65b2f4b6ced983db6f3542d8e59ba2d40ebacec5f082591
Sep 16 12:56:01.335745 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:01.335705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m" event={"ID":"e742f57c-4756-49fc-a306-2d69e9107678","Type":"ContainerStarted","Data":"da47666b55217a77a65b2f4b6ced983db6f3542d8e59ba2d40ebacec5f082591"}
Sep 16 12:56:04.356296 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:04.356261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m" event={"ID":"e742f57c-4756-49fc-a306-2d69e9107678","Type":"ContainerStarted","Data":"d7554c87c5a008e5035d267e60bcd7b90af281bb33ff7f3846b21bf10f251713"}
Sep 16 12:56:04.356701 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:04.356330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:04.377944 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:04.377886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m" podStartSLOduration=1.666896356 podStartE2EDuration="4.377870032s" totalImagesPullingTime="2.710973676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:04.37623844 +0000 UTC m=+470.415037562" watchObservedRunningTime="2026-09-16 12:56:04.377870032 +0000 UTC m=+470.416669154"
Sep 16 12:56:15.363110 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:15.363079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cc7b8d69b-h629m"
Sep 16 12:56:42.572223 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.572184    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"]
Sep 16 12:56:42.574151 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.574136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.578944 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.578913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:56:42.578944 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.578914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 12:56:42.579130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.578913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:56:42.579130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.578914    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 12:56:42.579130 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.578924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-54rsc\""
Sep 16 12:56:42.590838 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.590810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"]
Sep 16 12:56:42.668773 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.668735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.668951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.668777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwvt\" (UniqueName: \"kubernetes.io/projected/e5d00af6-0c0e-48c3-a01a-af1638ff891d-kube-api-access-tgwvt\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.668951 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.668805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5d00af6-0c0e-48c3-a01a-af1638ff891d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.769534 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.769498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.769746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.769544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwvt\" (UniqueName: \"kubernetes.io/projected/e5d00af6-0c0e-48c3-a01a-af1638ff891d-kube-api-access-tgwvt\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.769746 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.769679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5d00af6-0c0e-48c3-a01a-af1638ff891d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.769861 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:56:42.769680    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 12:56:42.769861 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:56:42.769853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert podName:e5d00af6-0c0e-48c3-a01a-af1638ff891d nodeName:}" failed. No retries permitted until 2026-09-16 12:56:43.269833887 +0000 UTC m=+509.308633006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-d2zkc" (UID: "e5d00af6-0c0e-48c3-a01a-af1638ff891d") : secret "plugin-serving-cert" not found
Sep 16 12:56:42.770261 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.770242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5d00af6-0c0e-48c3-a01a-af1638ff891d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:42.780353 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:42.780325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwvt\" (UniqueName: \"kubernetes.io/projected/e5d00af6-0c0e-48c3-a01a-af1638ff891d-kube-api-access-tgwvt\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:43.274480 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:43.274444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:43.276910 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:43.276879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d00af6-0c0e-48c3-a01a-af1638ff891d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d2zkc\" (UID: \"e5d00af6-0c0e-48c3-a01a-af1638ff891d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:43.483246 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:43.483213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"
Sep 16 12:56:43.754662 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:43.751937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc"]
Sep 16 12:56:43.757507 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:56:43.757480    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d00af6_0c0e_48c3_a01a_af1638ff891d.slice/crio-58b53e54b4ed7e320e8d00049525305a8c38f9e4440ffe5fb0d5561bf3fc05ff WatchSource:0}: Error finding container 58b53e54b4ed7e320e8d00049525305a8c38f9e4440ffe5fb0d5561bf3fc05ff: Status 404 returned error can't find the container with id 58b53e54b4ed7e320e8d00049525305a8c38f9e4440ffe5fb0d5561bf3fc05ff
Sep 16 12:56:44.509264 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:44.509229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc" event={"ID":"e5d00af6-0c0e-48c3-a01a-af1638ff891d","Type":"ContainerStarted","Data":"58b53e54b4ed7e320e8d00049525305a8c38f9e4440ffe5fb0d5561bf3fc05ff"}
Sep 16 12:56:48.533544 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:48.533506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc" event={"ID":"e5d00af6-0c0e-48c3-a01a-af1638ff891d","Type":"ContainerStarted","Data":"c3dc527737bf0ee57f9919f67db71ec423d0f2f72f1894a72b0ff52338b60866"}
Sep 16 12:56:48.549492 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:56:48.549444    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d2zkc" podStartSLOduration=1.8418205140000001 podStartE2EDuration="6.549425658s" totalImagesPullingTime="4.707605144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:48.549041344 +0000 UTC m=+514.587840467" watchObservedRunningTime="2026-09-16 12:56:48.549425658 +0000 UTC m=+514.588224781"
Sep 16 12:57:25.783591 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:25.783541    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:25.801834 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:25.801799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:25.802923 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:25.802899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:25.804695 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:25.804676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2vz22\""
Sep 16 12:57:25.934131 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:25.934090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrjm\" (UniqueName: \"kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm\") pod \"authorino-7bcfd8f9cd-7lwgj\" (UID: \"d8385516-a35e-41f5-94bb-cba44e244099\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:26.035218 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.035130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrjm\" (UniqueName: \"kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm\") pod \"authorino-7bcfd8f9cd-7lwgj\" (UID: \"d8385516-a35e-41f5-94bb-cba44e244099\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:26.058362 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.058332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrjm\" (UniqueName: \"kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm\") pod \"authorino-7bcfd8f9cd-7lwgj\" (UID: \"d8385516-a35e-41f5-94bb-cba44e244099\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:26.084508 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.084475    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:57:26.087617 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.087596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:57:26.102341 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.102316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:57:26.112041 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.112016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:26.237267 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.237235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d55x\" (UniqueName: \"kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x\") pod \"authorino-59c974bd48-2vwx7\" (UID: \"dc59e0a3-fce6-449c-9386-cac4c28148db\") " pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:57:26.338773 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.338695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d55x\" (UniqueName: \"kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x\") pod \"authorino-59c974bd48-2vwx7\" (UID: \"dc59e0a3-fce6-449c-9386-cac4c28148db\") " pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:57:26.352154 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.352126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d55x\" (UniqueName: \"kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x\") pod \"authorino-59c974bd48-2vwx7\" (UID: \"dc59e0a3-fce6-449c-9386-cac4c28148db\") " pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:57:26.393696 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.393668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:26.395949 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.395928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:57:26.396297 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:57:26.396270    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8385516_a35e_41f5_94bb_cba44e244099.slice/crio-9c03b4b2caef4fb63354ec7b8e00ccd5c6be1c3d189d918a14a4ac22b3eb239e WatchSource:0}: Error finding container 9c03b4b2caef4fb63354ec7b8e00ccd5c6be1c3d189d918a14a4ac22b3eb239e: Status 404 returned error can't find the container with id 9c03b4b2caef4fb63354ec7b8e00ccd5c6be1c3d189d918a14a4ac22b3eb239e
Sep 16 12:57:26.676931 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.676899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:57:26.680190 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:57:26.680161    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc59e0a3_fce6_449c_9386_cac4c28148db.slice/crio-e843347e8232b16784bab2651beb462b36e4985f3743db67188084a3d1285f4f WatchSource:0}: Error finding container e843347e8232b16784bab2651beb462b36e4985f3743db67188084a3d1285f4f: Status 404 returned error can't find the container with id e843347e8232b16784bab2651beb462b36e4985f3743db67188084a3d1285f4f
Sep 16 12:57:26.687335 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.687308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" event={"ID":"d8385516-a35e-41f5-94bb-cba44e244099","Type":"ContainerStarted","Data":"9c03b4b2caef4fb63354ec7b8e00ccd5c6be1c3d189d918a14a4ac22b3eb239e"}
Sep 16 12:57:26.688413 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:26.688393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2vwx7" event={"ID":"dc59e0a3-fce6-449c-9386-cac4c28148db","Type":"ContainerStarted","Data":"e843347e8232b16784bab2651beb462b36e4985f3743db67188084a3d1285f4f"}
Sep 16 12:57:31.712152 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:31.712114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2vwx7" event={"ID":"dc59e0a3-fce6-449c-9386-cac4c28148db","Type":"ContainerStarted","Data":"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"}
Sep 16 12:57:31.735925 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:31.735872    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2vwx7" podStartSLOduration=1.324223294 podStartE2EDuration="5.735856941s" totalImagesPullingTime="4.411633647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:31.735516951 +0000 UTC m=+557.774316073" watchObservedRunningTime="2026-09-16 12:57:31.735856941 +0000 UTC m=+557.774656063"
Sep 16 12:57:31.789910 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:31.789877    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:38.740281 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:38.740237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" event={"ID":"d8385516-a35e-41f5-94bb-cba44e244099","Type":"ContainerStarted","Data":"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"}
Sep 16 12:57:38.740720 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:38.740296    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" podUID="d8385516-a35e-41f5-94bb-cba44e244099" containerName="authorino" containerID="cri-o://c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802" gracePeriod=30
Sep 16 12:57:38.765026 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:38.764971    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" podStartSLOduration=2.3841601040000002 podStartE2EDuration="13.764956376s" totalImagesPullingTime="11.380796272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:38.764448737 +0000 UTC m=+564.803247860" watchObservedRunningTime="2026-09-16 12:57:38.764956376 +0000 UTC m=+564.803755552"
Sep 16 12:57:38.976759 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:38.976736    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:39.041252 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.041176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rrjm\" (UniqueName: \"kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm\") pod \"d8385516-a35e-41f5-94bb-cba44e244099\" (UID: \"d8385516-a35e-41f5-94bb-cba44e244099\") "
Sep 16 12:57:39.043374 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.043346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm" (OuterVolumeSpecName: "kube-api-access-6rrjm") pod "d8385516-a35e-41f5-94bb-cba44e244099" (UID: "d8385516-a35e-41f5-94bb-cba44e244099"). InnerVolumeSpecName "kube-api-access-6rrjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:39.142236 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.142206    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rrjm\" (UniqueName: \"kubernetes.io/projected/d8385516-a35e-41f5-94bb-cba44e244099-kube-api-access-6rrjm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:57:39.745783 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.745748    2579 generic.go:356] "Generic (PLEG): container finished" podID="d8385516-a35e-41f5-94bb-cba44e244099" containerID="c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802" exitCode=0
Sep 16 12:57:39.746225 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.745788    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj"
Sep 16 12:57:39.746225 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.745836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" event={"ID":"d8385516-a35e-41f5-94bb-cba44e244099","Type":"ContainerDied","Data":"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"}
Sep 16 12:57:39.746225 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.745873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7lwgj" event={"ID":"d8385516-a35e-41f5-94bb-cba44e244099","Type":"ContainerDied","Data":"9c03b4b2caef4fb63354ec7b8e00ccd5c6be1c3d189d918a14a4ac22b3eb239e"}
Sep 16 12:57:39.746225 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.745891    2579 scope.go:117] "RemoveContainer" containerID="c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"
Sep 16 12:57:39.754650 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.754635    2579 scope.go:117] "RemoveContainer" containerID="c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"
Sep 16 12:57:39.754904 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:57:39.754885    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802\": container with ID starting with c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802 not found: ID does not exist" containerID="c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"
Sep 16 12:57:39.754953 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.754910    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802"} err="failed to get container status \"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802\": rpc error: code = NotFound desc = could not find container \"c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802\": container with ID starting with c924942912304d294420756970b51566ab0c3e43b16319bb835f1ca3e7144802 not found: ID does not exist"
Sep 16 12:57:39.768208 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.768185    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:39.772147 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:39.772124    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7lwgj"]
Sep 16 12:57:40.457205 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:57:40.457162    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8385516-a35e-41f5-94bb-cba44e244099" path="/var/lib/kubelet/pods/d8385516-a35e-41f5-94bb-cba44e244099/volumes"
Sep 16 12:58:00.289376 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.289344    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-mtr9c"]
Sep 16 12:58:00.289833 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.289620    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8385516-a35e-41f5-94bb-cba44e244099" containerName="authorino"
Sep 16 12:58:00.289833 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.289631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8385516-a35e-41f5-94bb-cba44e244099" containerName="authorino"
Sep 16 12:58:00.289833 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.289689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8385516-a35e-41f5-94bb-cba44e244099" containerName="authorino"
Sep 16 12:58:00.293931 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.293915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.296892 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.296873    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:58:00.302127 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.302108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-mtr9c"]
Sep 16 12:58:00.388385 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.388354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx52k\" (UniqueName: \"kubernetes.io/projected/4f1508c2-2d2a-4055-b0ad-20da138f0a78-kube-api-access-jx52k\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.388529 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.388400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f1508c2-2d2a-4055-b0ad-20da138f0a78-tls-cert\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.489618 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.489594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx52k\" (UniqueName: \"kubernetes.io/projected/4f1508c2-2d2a-4055-b0ad-20da138f0a78-kube-api-access-jx52k\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.489759 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.489637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f1508c2-2d2a-4055-b0ad-20da138f0a78-tls-cert\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.491862 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.491842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f1508c2-2d2a-4055-b0ad-20da138f0a78-tls-cert\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.499313 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.499294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx52k\" (UniqueName: \"kubernetes.io/projected/4f1508c2-2d2a-4055-b0ad-20da138f0a78-kube-api-access-jx52k\") pod \"authorino-6d9898bd99-mtr9c\" (UID: \"4f1508c2-2d2a-4055-b0ad-20da138f0a78\") " pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.602662 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.602600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-mtr9c"
Sep 16 12:58:00.874133 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:00.874051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-mtr9c"]
Sep 16 12:58:00.877896 ip-10-0-137-128 kubenswrapper[2579]: W0916 12:58:00.877870    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f1508c2_2d2a_4055_b0ad_20da138f0a78.slice/crio-d665e6a0b0c847222dbabd2030c3e693c04478968cc5df812a88aea4898300a0 WatchSource:0}: Error finding container d665e6a0b0c847222dbabd2030c3e693c04478968cc5df812a88aea4898300a0: Status 404 returned error can't find the container with id d665e6a0b0c847222dbabd2030c3e693c04478968cc5df812a88aea4898300a0
Sep 16 12:58:01.837079 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:01.837046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-mtr9c" event={"ID":"4f1508c2-2d2a-4055-b0ad-20da138f0a78","Type":"ContainerStarted","Data":"d665e6a0b0c847222dbabd2030c3e693c04478968cc5df812a88aea4898300a0"}
Sep 16 12:58:03.847171 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:03.847135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-mtr9c" event={"ID":"4f1508c2-2d2a-4055-b0ad-20da138f0a78","Type":"ContainerStarted","Data":"c6ea9d936c28a0ed1b111a41bef2ee18e1db298bd9cab16767663f01d845d0e1"}
Sep 16 12:58:03.879149 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:03.879103    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-mtr9c" podStartSLOduration=1.007761278 podStartE2EDuration="3.879088368s" totalImagesPullingTime="2.87132709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:03.876820122 +0000 UTC m=+589.915619246" watchObservedRunningTime="2026-09-16 12:58:03.879088368 +0000 UTC m=+589.917887489"
Sep 16 12:58:03.920721 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:03.920680    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:58:03.920906 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:03.920882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2vwx7" podUID="dc59e0a3-fce6-449c-9386-cac4c28148db" containerName="authorino" containerID="cri-o://77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328" gracePeriod=30
Sep 16 12:58:04.157280 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.157244    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:58:04.217205 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.217176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d55x\" (UniqueName: \"kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x\") pod \"dc59e0a3-fce6-449c-9386-cac4c28148db\" (UID: \"dc59e0a3-fce6-449c-9386-cac4c28148db\") "
Sep 16 12:58:04.219080 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.219059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x" (OuterVolumeSpecName: "kube-api-access-8d55x") pod "dc59e0a3-fce6-449c-9386-cac4c28148db" (UID: "dc59e0a3-fce6-449c-9386-cac4c28148db"). InnerVolumeSpecName "kube-api-access-8d55x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:04.317961 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.317901    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d55x\" (UniqueName: \"kubernetes.io/projected/dc59e0a3-fce6-449c-9386-cac4c28148db-kube-api-access-8d55x\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 12:58:04.852323 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.852285    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc59e0a3-fce6-449c-9386-cac4c28148db" containerID="77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328" exitCode=0
Sep 16 12:58:04.852323 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.852321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2vwx7"
Sep 16 12:58:04.852762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.852333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2vwx7" event={"ID":"dc59e0a3-fce6-449c-9386-cac4c28148db","Type":"ContainerDied","Data":"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"}
Sep 16 12:58:04.852762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.852374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2vwx7" event={"ID":"dc59e0a3-fce6-449c-9386-cac4c28148db","Type":"ContainerDied","Data":"e843347e8232b16784bab2651beb462b36e4985f3743db67188084a3d1285f4f"}
Sep 16 12:58:04.852762 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.852390    2579 scope.go:117] "RemoveContainer" containerID="77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"
Sep 16 12:58:04.860083 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.860066    2579 scope.go:117] "RemoveContainer" containerID="77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"
Sep 16 12:58:04.860336 ip-10-0-137-128 kubenswrapper[2579]: E0916 12:58:04.860317    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328\": container with ID starting with 77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328 not found: ID does not exist" containerID="77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"
Sep 16 12:58:04.860383 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.860347    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328"} err="failed to get container status \"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328\": rpc error: code = NotFound desc = could not find container \"77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328\": container with ID starting with 77c474657caac089b6c137b98c43756bf90cb52940f135ccc842c999df35f328 not found: ID does not exist"
Sep 16 12:58:04.871832 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.871809    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:58:04.877435 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:04.877415    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2vwx7"]
Sep 16 12:58:06.458027 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:06.457991    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc59e0a3-fce6-449c-9386-cac4c28148db" path="/var/lib/kubelet/pods/dc59e0a3-fce6-449c-9386-cac4c28148db/volumes"
Sep 16 12:58:14.354462 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:14.354432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 12:58:14.357163 ip-10-0-137-128 kubenswrapper[2579]: I0916 12:58:14.357143    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:02:13.454882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.454853    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": read tcp 10.134.0.2:42680->10.134.0.21:15021: read: connection reset by peer" start-of-body=
Sep 16 13:02:13.455281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.454925    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": read tcp 10.134.0.2:42680->10.134.0.21:15021: read: connection reset by peer"
Sep 16 13:02:13.845455 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.845373    2579 generic.go:356] "Generic (PLEG): container finished" podID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerID="bc511f150a007fd47ab5d1db32470894f0719570fd80e3d281a580e4ef71f8c7" exitCode=0
Sep 16 13:02:13.845455 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.845430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" event={"ID":"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75","Type":"ContainerDied","Data":"bc511f150a007fd47ab5d1db32470894f0719570fd80e3d281a580e4ef71f8c7"}
Sep 16 13:02:13.845823 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.845810    2579 scope.go:117] "RemoveContainer" containerID="bc511f150a007fd47ab5d1db32470894f0719570fd80e3d281a580e4ef71f8c7"
Sep 16 13:02:13.846764 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.846735    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:02:13.846842 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.846828    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:02:13.846877 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:13.846866    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:02:14.851882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:14.851845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" event={"ID":"2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75","Type":"ContainerStarted","Data":"375807158cde2449873efa033d040406b404cea842e4679e24ef08c72fa17bd0"}
Sep 16 13:02:15.050206 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:15.050172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 13:02:15.050475 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:15.050451    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Sep 16 13:02:15.050525 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:15.050502    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Sep 16 13:02:16.050025 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:16.049991    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Sep 16 13:02:16.050405 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:16.050046    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Sep 16 13:02:17.050114 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:17.050070    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Sep 16 13:02:17.050603 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:17.050133    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Sep 16 13:02:18.050437 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:18.050400    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Sep 16 13:02:18.050829 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:18.050453    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Sep 16 13:02:19.050382 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:19.050346    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Sep 16 13:02:19.050576 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:19.050417    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq" podUID="2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Sep 16 13:02:20.053246 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:20.053218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 13:02:20.053670 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:20.053457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 13:02:20.054075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:20.054059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq"
Sep 16 13:02:31.591985 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.591939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:02:31.592575 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.592378    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc59e0a3-fce6-449c-9386-cac4c28148db" containerName="authorino"
Sep 16 13:02:31.592575 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.592398    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc59e0a3-fce6-449c-9386-cac4c28148db" containerName="authorino"
Sep 16 13:02:31.592575 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.592478    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc59e0a3-fce6-449c-9386-cac4c28148db" containerName="authorino"
Sep 16 13:02:31.595696 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.595673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.599695 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.599672    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-f8nh7\""
Sep 16 13:02:31.599817 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.599702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 13:02:31.599817 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.599675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 13:02:31.599817 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.599725    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 13:02:31.610004 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.609983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:02:31.751125 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frg2d\" (UniqueName: \"kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.751275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.751275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.751275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.751441 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.751441 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.751357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852133 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852133 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852133 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg2d\" (UniqueName: \"kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852547 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852635 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.852674 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.852646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.854236 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.854216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.854466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.854449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.870232 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.870207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frg2d\" (UniqueName: \"kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d\") pod \"fc-smoke-test-kserve-7f4687897-dccsl\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.907237 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.907211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:31.938274 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.938246    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:02:31.941608 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.941591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:31.944230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.944206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-92hq7\""
Sep 16 13:02:31.951314 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:31.951287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:02:32.053464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.053435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.053464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.053471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzszk\" (UniqueName: \"kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.154338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.154301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.154338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.154340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzszk\" (UniqueName: \"kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.156777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.156748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.162334 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.162293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzszk\" (UniqueName: \"kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.191636 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.191420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:02:32.193822 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:02:32.193797    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcbced3_0723_448e_a7f8_5706182f0a2c.slice/crio-5e643f94e63e50ff56e5bb2b2e5631b5bfa38d7f56afe5aee258e6e2a0f22a44 WatchSource:0}: Error finding container 5e643f94e63e50ff56e5bb2b2e5631b5bfa38d7f56afe5aee258e6e2a0f22a44: Status 404 returned error can't find the container with id 5e643f94e63e50ff56e5bb2b2e5631b5bfa38d7f56afe5aee258e6e2a0f22a44
Sep 16 13:02:32.195840 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.195822    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:02:32.253516 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.253490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:32.522819 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.522750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:02:32.525542 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:02:32.525515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9053a5e9_7c86_4896_a537_943b97b78577.slice/crio-86767d52b13017aece30d4924a43a189402c014bcb8a5fa165af41c4f35d0b94 WatchSource:0}: Error finding container 86767d52b13017aece30d4924a43a189402c014bcb8a5fa165af41c4f35d0b94: Status 404 returned error can't find the container with id 86767d52b13017aece30d4924a43a189402c014bcb8a5fa165af41c4f35d0b94
Sep 16 13:02:32.937808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.937758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" event={"ID":"9053a5e9-7c86-4896-a537-943b97b78577","Type":"ContainerStarted","Data":"86767d52b13017aece30d4924a43a189402c014bcb8a5fa165af41c4f35d0b94"}
Sep 16 13:02:32.940555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:32.940451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" event={"ID":"8dcbced3-0723-448e-a7f8-5706182f0a2c","Type":"ContainerStarted","Data":"5e643f94e63e50ff56e5bb2b2e5631b5bfa38d7f56afe5aee258e6e2a0f22a44"}
Sep 16 13:02:34.956072 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:34.956033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" event={"ID":"8dcbced3-0723-448e-a7f8-5706182f0a2c","Type":"ContainerStarted","Data":"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"}
Sep 16 13:02:34.976892 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:34.976829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" podStartSLOduration=2.172145832 podStartE2EDuration="3.976810377s" totalImagesPullingTime="1.804664545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:34.974676357 +0000 UTC m=+861.013475481" watchObservedRunningTime="2026-09-16 13:02:34.976810377 +0000 UTC m=+861.015609501"
Sep 16 13:02:35.907499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:35.907455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:35.966686 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:35.966649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" event={"ID":"9053a5e9-7c86-4896-a537-943b97b78577","Type":"ContainerStarted","Data":"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"}
Sep 16 13:02:35.984397 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:35.984349    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podStartSLOduration=2.322151664 podStartE2EDuration="4.984334795s" totalImagesPullingTime="2.662183131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:35.98388431 +0000 UTC m=+862.022683432" watchObservedRunningTime="2026-09-16 13:02:35.984334795 +0000 UTC m=+862.023133932"
Sep 16 13:02:36.970521 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:36.970488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:02:41.907578 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:41.907518    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:41.920134 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:41.920104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:02:42.003011 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:02:42.002980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:03:07.976545 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:07.976466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:03:14.379513 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.379481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:03:14.381534 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.381510    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:03:14.587238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.587201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:03:14.599743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.599706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:03:14.607972 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:03:14.607931    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:03:14.607972 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:03:14.607972 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:03:14.607972 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e" cmd=["/bin/sleep","15"]
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.607993    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" message="" err=<
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:03:14.608004    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerName="main" containerID="cri-o://f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"
Sep 16 13:03:14.608242 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.608046    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerName="main" containerID="cri-o://f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e" gracePeriod=60
Sep 16 13:03:14.949555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:14.949532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:03:15.021595 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021595 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021595 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021595 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frg2d\" (UniqueName: \"kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d\") pod \"8dcbced3-0723-448e-a7f8-5706182f0a2c\" (UID: \"8dcbced3-0723-448e-a7f8-5706182f0a2c\") "
Sep 16 13:03:15.021907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home" (OuterVolumeSpecName: "home") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:15.021907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:15.022055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.021916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:15.023908 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.023883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:15.024030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.023926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm" (OuterVolumeSpecName: "dshm") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:15.024030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.023922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d" (OuterVolumeSpecName: "kube-api-access-frg2d") pod "8dcbced3-0723-448e-a7f8-5706182f0a2c" (UID: "8dcbced3-0723-448e-a7f8-5706182f0a2c"). InnerVolumeSpecName "kube-api-access-frg2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:15.123022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.122992    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.123022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.123017    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.123022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.123028    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcbced3-0723-448e-a7f8-5706182f0a2c-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.123281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.123036    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.123281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.123043    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcbced3-0723-448e-a7f8-5706182f0a2c-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.123281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.123051    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frg2d\" (UniqueName: \"kubernetes.io/projected/8dcbced3-0723-448e-a7f8-5706182f0a2c-kube-api-access-frg2d\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:15.140787 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.140752    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerID="f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e" exitCode=0
Sep 16 13:03:15.140970 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.140805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"
Sep 16 13:03:15.140970 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.140835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" event={"ID":"8dcbced3-0723-448e-a7f8-5706182f0a2c","Type":"ContainerDied","Data":"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"}
Sep 16 13:03:15.140970 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.140905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl" event={"ID":"8dcbced3-0723-448e-a7f8-5706182f0a2c","Type":"ContainerDied","Data":"5e643f94e63e50ff56e5bb2b2e5631b5bfa38d7f56afe5aee258e6e2a0f22a44"}
Sep 16 13:03:15.140970 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.140937    2579 scope.go:117] "RemoveContainer" containerID="f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"
Sep 16 13:03:15.150820 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.150800    2579 scope.go:117] "RemoveContainer" containerID="f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"
Sep 16 13:03:15.151195 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:03:15.151160    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e\": container with ID starting with f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e not found: ID does not exist" containerID="f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"
Sep 16 13:03:15.151662 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.151619    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e"} err="failed to get container status \"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e\": rpc error: code = NotFound desc = could not find container \"f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e\": container with ID starting with f755fc1c6ccd205512a6412cf77b05fba25683c1a8c46ea1fd0283a935ebfc1e not found: ID does not exist"
Sep 16 13:03:15.211693 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.211660    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:03:15.212834 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:15.212812    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-dccsl"]
Sep 16 13:03:16.457871 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:16.457841    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" path="/var/lib/kubelet/pods/8dcbced3-0723-448e-a7f8-5706182f0a2c/volumes"
Sep 16 13:03:17.976806 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:17.976766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:03:27.976598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:27.976543    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:03:29.608346 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:29.608310    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" containerID="cri-o://acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d" gracePeriod=45
Sep 16 13:03:37.976168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:37.976124    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:03:37.976541 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:37.976229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:03:47.976116 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:47.976076    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:03:57.976352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:57.976307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:03:59.848303 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.848283    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:03:59.874733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.874677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzszk\" (UniqueName: \"kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk\") pod \"9053a5e9-7c86-4896-a537-943b97b78577\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") "
Sep 16 13:03:59.874733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.874732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs\") pod \"9053a5e9-7c86-4896-a537-943b97b78577\" (UID: \"9053a5e9-7c86-4896-a537-943b97b78577\") "
Sep 16 13:03:59.876651 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.876619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk" (OuterVolumeSpecName: "kube-api-access-zzszk") pod "9053a5e9-7c86-4896-a537-943b97b78577" (UID: "9053a5e9-7c86-4896-a537-943b97b78577"). InnerVolumeSpecName "kube-api-access-zzszk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:59.876750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.876681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9053a5e9-7c86-4896-a537-943b97b78577" (UID: "9053a5e9-7c86-4896-a537-943b97b78577"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:59.976231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.976201    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzszk\" (UniqueName: \"kubernetes.io/projected/9053a5e9-7c86-4896-a537-943b97b78577-kube-api-access-zzszk\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:03:59.976231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:03:59.976226    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9053a5e9-7c86-4896-a537-943b97b78577-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:04:00.327834 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.327803    2579 generic.go:356] "Generic (PLEG): container finished" podID="9053a5e9-7c86-4896-a537-943b97b78577" containerID="acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d" exitCode=0
Sep 16 13:04:00.328006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.327845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"
Sep 16 13:04:00.328006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.327886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" event={"ID":"9053a5e9-7c86-4896-a537-943b97b78577","Type":"ContainerDied","Data":"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"}
Sep 16 13:04:00.328006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.327939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf" event={"ID":"9053a5e9-7c86-4896-a537-943b97b78577","Type":"ContainerDied","Data":"86767d52b13017aece30d4924a43a189402c014bcb8a5fa165af41c4f35d0b94"}
Sep 16 13:04:00.328006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.327969    2579 scope.go:117] "RemoveContainer" containerID="acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"
Sep 16 13:04:00.336706 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.336692    2579 scope.go:117] "RemoveContainer" containerID="acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"
Sep 16 13:04:00.336964 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:04:00.336947    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d\": container with ID starting with acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d not found: ID does not exist" containerID="acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"
Sep 16 13:04:00.337015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.336972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d"} err="failed to get container status \"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d\": rpc error: code = NotFound desc = could not find container \"acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d\": container with ID starting with acc8d2aeedf8a753d2ca2f6cc96103f550278ae59ca9a34061bf7be6cd821e7d not found: ID does not exist"
Sep 16 13:04:00.348710 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.348689    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:04:00.352197 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.352177    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-trlzf"]
Sep 16 13:04:00.458046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:00.458012    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9053a5e9-7c86-4896-a537-943b97b78577" path="/var/lib/kubelet/pods/9053a5e9-7c86-4896-a537-943b97b78577/volumes"
Sep 16 13:04:20.990664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main"
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990913    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main"
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990929    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerName="main"
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerName="main"
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.990991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dcbced3-0723-448e-a7f8-5706182f0a2c" containerName="main"
Sep 16 13:04:20.991127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.991000    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9053a5e9-7c86-4896-a537-943b97b78577" containerName="main"
Sep 16 13:04:20.995025 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.995005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:20.998817 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.998792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-64pqx\""
Sep 16 13:04:20.998948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.998803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 13:04:20.998948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.998849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 13:04:20.998948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:20.998833    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 13:04:21.005292 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.005270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:04:21.136072 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.136268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.136268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfcp\" (UniqueName: \"kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.136268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.136268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.136464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.136274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfcp\" (UniqueName: \"kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237845 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237845 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.237903 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.237879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.238200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.238178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.238270 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.238214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.238270 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.238247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.240193 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.240160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.240193 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.240183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.240901 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.240858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:04:21.244050 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.244036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.246371 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.246350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-hhsjc\""
Sep 16 13:04:21.248505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.248487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfcp\" (UniqueName: \"kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp\") pod \"fc-concurrency-test-kserve-b5d44467c-xw45n\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.252633 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.252615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:04:21.306304 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.306279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:21.338865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.338833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.339019 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.338905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn94\" (UniqueName: \"kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.440325 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.440291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.440495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.440353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn94\" (UniqueName: \"kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.442629 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.442610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.448110 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.448069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn94\" (UniqueName: \"kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.567751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.567682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:21.578264 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.578245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:04:21.580606 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:04:21.580585    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7678b049_161d_4a3c_84cf_06525c78d514.slice/crio-b98cd30b194b71b8538cb5646f79746f41a7c556f7616cd5c9462f60dfbfd1ee WatchSource:0}: Error finding container b98cd30b194b71b8538cb5646f79746f41a7c556f7616cd5c9462f60dfbfd1ee: Status 404 returned error can't find the container with id b98cd30b194b71b8538cb5646f79746f41a7c556f7616cd5c9462f60dfbfd1ee
Sep 16 13:04:21.837981 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:21.837887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:04:21.839824 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:04:21.839801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed2260f_37e4_4d57_9177_cbce01fa683c.slice/crio-9f0da47737e8dc9d47c9fa73e68b16f576d7d7fc269813994eef32388366046f WatchSource:0}: Error finding container 9f0da47737e8dc9d47c9fa73e68b16f576d7d7fc269813994eef32388366046f: Status 404 returned error can't find the container with id 9f0da47737e8dc9d47c9fa73e68b16f576d7d7fc269813994eef32388366046f
Sep 16 13:04:22.424496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.424458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" event={"ID":"7678b049-161d-4a3c-84cf-06525c78d514","Type":"ContainerStarted","Data":"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"}
Sep 16 13:04:22.424945 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.424518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" event={"ID":"7678b049-161d-4a3c-84cf-06525c78d514","Type":"ContainerStarted","Data":"b98cd30b194b71b8538cb5646f79746f41a7c556f7616cd5c9462f60dfbfd1ee"}
Sep 16 13:04:22.426827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.426305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" event={"ID":"eed2260f-37e4-4d57-9177-cbce01fa683c","Type":"ContainerStarted","Data":"6e5965d8f9f9d8b20e31d93e3870e94be0094eda39c240a50bfc356294ad3798"}
Sep 16 13:04:22.426827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.426361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:22.426827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.426388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" event={"ID":"eed2260f-37e4-4d57-9177-cbce01fa683c","Type":"ContainerStarted","Data":"9f0da47737e8dc9d47c9fa73e68b16f576d7d7fc269813994eef32388366046f"}
Sep 16 13:04:22.466458 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.466395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" podStartSLOduration=2.466373881 podStartE2EDuration="2.466373881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:22.444607114 +0000 UTC m=+968.483406234" watchObservedRunningTime="2026-09-16 13:04:22.466373881 +0000 UTC m=+968.505173006"
Sep 16 13:04:22.467652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:22.467606    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podStartSLOduration=1.4675908739999999 podStartE2EDuration="1.467590874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:04:22.465881188 +0000 UTC m=+968.504680313" watchObservedRunningTime="2026-09-16 13:04:22.467590874 +0000 UTC m=+968.506389998"
Sep 16 13:04:23.306862 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:23.306831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:31.307346 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:31.307315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:31.319855 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:31.319831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:31.473425 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:31.473400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:04:53.432364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:53.432281    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:04:59.760992 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:59.760962    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:04:59.769502 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:04:59.769460    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:04:59.769502 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:04:59.769502 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:04:59.769502 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892" cmd=["/bin/sleep","15"]
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:59.769525    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" message="" err=<
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:04:59.769540    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" podUID="7678b049-161d-4a3c-84cf-06525c78d514" containerName="main" containerID="cri-o://cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"
Sep 16 13:04:59.769849 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:59.769626    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" podUID="7678b049-161d-4a3c-84cf-06525c78d514" containerName="main" containerID="cri-o://cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892" gracePeriod=60
Sep 16 13:04:59.773188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:04:59.773166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:05:00.112735 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.112709    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:05:00.166051 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166051 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166053    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166259 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166259 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166225    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166370 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166370 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gfcp\" (UniqueName: \"kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp\") pod \"7678b049-161d-4a3c-84cf-06525c78d514\" (UID: \"7678b049-161d-4a3c-84cf-06525c78d514\") "
Sep 16 13:05:00.166501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home" (OuterVolumeSpecName: "home") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:00.166575 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache" (OuterVolumeSpecName: "model-cache") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:00.166635 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:00.166635 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166615    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.166735 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.166639    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.168203 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.168176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm" (OuterVolumeSpecName: "dshm") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:00.168304 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.168260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp" (OuterVolumeSpecName: "kube-api-access-2gfcp") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "kube-api-access-2gfcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:00.168392 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.168377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7678b049-161d-4a3c-84cf-06525c78d514" (UID: "7678b049-161d-4a3c-84cf-06525c78d514"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:00.267758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.267726    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.267758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.267752    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7678b049-161d-4a3c-84cf-06525c78d514-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.267758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.267762    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfcp\" (UniqueName: \"kubernetes.io/projected/7678b049-161d-4a3c-84cf-06525c78d514-kube-api-access-2gfcp\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.267955 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.267770    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7678b049-161d-4a3c-84cf-06525c78d514-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:00.589766 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.589734    2579 generic.go:356] "Generic (PLEG): container finished" podID="7678b049-161d-4a3c-84cf-06525c78d514" containerID="cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892" exitCode=0
Sep 16 13:05:00.589932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.589782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"
Sep 16 13:05:00.589932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.589816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" event={"ID":"7678b049-161d-4a3c-84cf-06525c78d514","Type":"ContainerDied","Data":"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"}
Sep 16 13:05:00.589932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.589867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n" event={"ID":"7678b049-161d-4a3c-84cf-06525c78d514","Type":"ContainerDied","Data":"b98cd30b194b71b8538cb5646f79746f41a7c556f7616cd5c9462f60dfbfd1ee"}
Sep 16 13:05:00.589932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.589889    2579 scope.go:117] "RemoveContainer" containerID="cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"
Sep 16 13:05:00.598195 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.598179    2579 scope.go:117] "RemoveContainer" containerID="cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"
Sep 16 13:05:00.598437 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:05:00.598422    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892\": container with ID starting with cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892 not found: ID does not exist" containerID="cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"
Sep 16 13:05:00.598483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.598444    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892"} err="failed to get container status \"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892\": rpc error: code = NotFound desc = could not find container \"cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892\": container with ID starting with cbf3c34a0f491ef1dc10d5b137900f6344264f2cb5be9e75a045651be554c892 not found: ID does not exist"
Sep 16 13:05:00.606979 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.606957    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:05:00.612281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:00.612260    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-xw45n"]
Sep 16 13:05:02.457419 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:02.457385    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7678b049-161d-4a3c-84cf-06525c78d514" path="/var/lib/kubelet/pods/7678b049-161d-4a3c-84cf-06525c78d514/volumes"
Sep 16 13:05:03.432501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:03.432458    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:05:13.178617 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.178557    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:05:13.178995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.178877    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7678b049-161d-4a3c-84cf-06525c78d514" containerName="main"
Sep 16 13:05:13.178995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.178888    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7678b049-161d-4a3c-84cf-06525c78d514" containerName="main"
Sep 16 13:05:13.178995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.178957    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7678b049-161d-4a3c-84cf-06525c78d514" containerName="main"
Sep 16 13:05:13.181995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.181976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.184588 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.184550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 13:05:13.184832 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.184816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-58zw5\""
Sep 16 13:05:13.185932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.185910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 13:05:13.186094 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.185989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 13:05:13.186094 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.185999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:05:13.186094 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.186008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 13:05:13.186238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.186125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-f7qtj\""
Sep 16 13:05:13.195143 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.195117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:05:13.272784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.272741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lksj\" (UniqueName: \"kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.272784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.272780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273037 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.272804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273037 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.272856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273037 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.272985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273037 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.273029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273237 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.273088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.273237 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.273136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374486 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lksj\" (UniqueName: \"kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.374697 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.375085 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.375085 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.374916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.375173 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.375149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.375211 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.375162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.375413 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.375395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.377039 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.377018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.377131 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.377062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.382915 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.382888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lksj\" (UniqueName: \"kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.432122 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.432029    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:05:13.492861 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.492823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:05:13.769273 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:13.769158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:05:13.775419 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:05:13.775316    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0271848_6c62_4321_8ae0_ca1ccc1f1e50.slice/crio-4e7e477f4270de27fea9c36086f4056bd77a07f216f5053845d1ba086bdd4acd WatchSource:0}: Error finding container 4e7e477f4270de27fea9c36086f4056bd77a07f216f5053845d1ba086bdd4acd: Status 404 returned error can't find the container with id 4e7e477f4270de27fea9c36086f4056bd77a07f216f5053845d1ba086bdd4acd
Sep 16 13:05:14.652040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:14.651991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerStarted","Data":"4e7e477f4270de27fea9c36086f4056bd77a07f216f5053845d1ba086bdd4acd"}
Sep 16 13:05:14.791067 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:14.790981    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" containerID="cri-o://6e5965d8f9f9d8b20e31d93e3870e94be0094eda39c240a50bfc356294ad3798" gracePeriod=45
Sep 16 13:05:17.109595 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:17.109549    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 13:05:17.668060 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:17.668025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerStarted","Data":"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"}
Sep 16 13:05:19.678763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:19.678676    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerID="6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f" exitCode=0
Sep 16 13:05:19.678763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:19.678751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerDied","Data":"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"}
Sep 16 13:05:23.432527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:23.432483    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:05:23.433102 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:23.432629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:05:33.432762 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:33.432718    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:05:43.432458 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:43.432413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:05:46.822956 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:46.822882    2579 generic.go:356] "Generic (PLEG): container finished" podID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerID="6e5965d8f9f9d8b20e31d93e3870e94be0094eda39c240a50bfc356294ad3798" exitCode=0
Sep 16 13:05:46.822956 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:46.822929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" event={"ID":"eed2260f-37e4-4d57-9177-cbce01fa683c","Type":"ContainerDied","Data":"6e5965d8f9f9d8b20e31d93e3870e94be0094eda39c240a50bfc356294ad3798"}
Sep 16 13:05:48.041818 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.041794    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:05:48.093646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.093614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxn94\" (UniqueName: \"kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94\") pod \"eed2260f-37e4-4d57-9177-cbce01fa683c\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") "
Sep 16 13:05:48.093793 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.093655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs\") pod \"eed2260f-37e4-4d57-9177-cbce01fa683c\" (UID: \"eed2260f-37e4-4d57-9177-cbce01fa683c\") "
Sep 16 13:05:48.095739 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.095711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eed2260f-37e4-4d57-9177-cbce01fa683c" (UID: "eed2260f-37e4-4d57-9177-cbce01fa683c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:48.095807 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.095730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94" (OuterVolumeSpecName: "kube-api-access-lxn94") pod "eed2260f-37e4-4d57-9177-cbce01fa683c" (UID: "eed2260f-37e4-4d57-9177-cbce01fa683c"). InnerVolumeSpecName "kube-api-access-lxn94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:48.194484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.194449    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxn94\" (UniqueName: \"kubernetes.io/projected/eed2260f-37e4-4d57-9177-cbce01fa683c-kube-api-access-lxn94\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:48.194484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.194481    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed2260f-37e4-4d57-9177-cbce01fa683c-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:05:48.837960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.837876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj" event={"ID":"eed2260f-37e4-4d57-9177-cbce01fa683c","Type":"ContainerDied","Data":"9f0da47737e8dc9d47c9fa73e68b16f576d7d7fc269813994eef32388366046f"}
Sep 16 13:05:48.837960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.837899    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"
Sep 16 13:05:48.837960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.837942    2579 scope.go:117] "RemoveContainer" containerID="6e5965d8f9f9d8b20e31d93e3870e94be0094eda39c240a50bfc356294ad3798"
Sep 16 13:05:48.840705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.840681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerStarted","Data":"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"}
Sep 16 13:05:48.855463 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.855438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:05:48.857934 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.857914    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-fpxxj"]
Sep 16 13:05:48.877100 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:48.877056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.5805788870000002 podStartE2EDuration="35.877041375s" totalImagesPullingTime="32.071531285s" totalInitContainerRuntime="2.224931203s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:48.875956864 +0000 UTC m=+1054.914755989" watchObservedRunningTime="2026-09-16 13:05:48.877041375 +0000 UTC m=+1054.915840498"
Sep 16 13:05:50.018108 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.018076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:05:50.018577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.018362    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main"
Sep 16 13:05:50.018577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.018373    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main"
Sep 16 13:05:50.018577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.018441    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" containerName="main"
Sep 16 13:05:50.021383 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.021361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.024187 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.024165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 13:05:50.024338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.024315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:05:50.024411 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.024335    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-wt9ps\""
Sep 16 13:05:50.024655 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.024632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 13:05:50.032374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.032351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:05:50.110044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwbw\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.110482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.110376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211284 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211284 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwbw\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211527 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.211858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.212022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.211906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.212133 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.212112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.212348 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.212328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.213858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.213842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.214465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.214445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.219379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.219348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.219971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.219951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwbw\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-whtbd\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.332931 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.332852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:50.458676 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.458643    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed2260f-37e4-4d57-9177-cbce01fa683c" path="/var/lib/kubelet/pods/eed2260f-37e4-4d57-9177-cbce01fa683c/volumes"
Sep 16 13:05:50.619987 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.619961    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:05:50.622216 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:05:50.622187    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b16750c_8fed_4c47_9322_a81eb0c9cf56.slice/crio-5e0f362669c3d25dd90247b5f3704423d8ce5fc3efe39f688878561d4e9cacee WatchSource:0}: Error finding container 5e0f362669c3d25dd90247b5f3704423d8ce5fc3efe39f688878561d4e9cacee: Status 404 returned error can't find the container with id 5e0f362669c3d25dd90247b5f3704423d8ce5fc3efe39f688878561d4e9cacee
Sep 16 13:05:50.624335 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.624298    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:05:50.624426 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.624367    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:05:50.624426 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.624398    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:05:50.853634 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.853590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" event={"ID":"4b16750c-8fed-4c47-9322-a81eb0c9cf56","Type":"ContainerStarted","Data":"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"}
Sep 16 13:05:50.853830 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.853657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" event={"ID":"4b16750c-8fed-4c47-9322-a81eb0c9cf56","Type":"ContainerStarted","Data":"5e0f362669c3d25dd90247b5f3704423d8ce5fc3efe39f688878561d4e9cacee"}
Sep 16 13:05:50.875764 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:50.875652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" podStartSLOduration=1.8756300380000002 podStartE2EDuration="1.875630038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:50.874294571 +0000 UTC m=+1056.913093693" watchObservedRunningTime="2026-09-16 13:05:50.875630038 +0000 UTC m=+1056.914429163"
Sep 16 13:05:51.333130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:51.333057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:51.338525 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:51.338500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:51.859246 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:51.859213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:05:51.860351 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:05:51.860320    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:06:04.028040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.028003    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:04.104523 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.104484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:04.104721 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.104645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.107715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.107683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-pz8nn\""
Sep 16 13:06:04.107844 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.107684    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 13:06:04.107844 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.107684    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 13:06:04.107844 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.107684    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzr8\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.238728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.233687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzr8\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.334786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.334720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.335115 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.335093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.336894 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.336875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.337482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.337458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.342427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.342400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.342767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.342750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzr8\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8\") pod \"router-gateway-1-openshift-default-66bbccc5fb-n49b5\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.415767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.415728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:04.747736 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.747691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:04.750747 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:06:04.750716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5794cc87_2aba_47e0_af29_672729ef1fa0.slice/crio-7d3cec32a3be36009bbd28a86229b2075bdf9972ae30100c5c9e5a2bfba45879 WatchSource:0}: Error finding container 7d3cec32a3be36009bbd28a86229b2075bdf9972ae30100c5c9e5a2bfba45879: Status 404 returned error can't find the container with id 7d3cec32a3be36009bbd28a86229b2075bdf9972ae30100c5c9e5a2bfba45879
Sep 16 13:06:04.752960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.752913    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:06:04.753065 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.753007    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:06:04.753065 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.753050    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 13:06:04.921889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.921796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" event={"ID":"5794cc87-2aba-47e0-af29-672729ef1fa0","Type":"ContainerStarted","Data":"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"}
Sep 16 13:06:04.921889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.921879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" event={"ID":"5794cc87-2aba-47e0-af29-672729ef1fa0","Type":"ContainerStarted","Data":"7d3cec32a3be36009bbd28a86229b2075bdf9972ae30100c5c9e5a2bfba45879"}
Sep 16 13:06:04.943730 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:04.943679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" podStartSLOduration=0.943663449 podStartE2EDuration="943.663449ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:04.942473311 +0000 UTC m=+1070.981272435" watchObservedRunningTime="2026-09-16 13:06:04.943663449 +0000 UTC m=+1070.982462572"
Sep 16 13:06:05.416415 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:05.416380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:05.417638 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:05.417608    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 13:06:06.420414 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:06.420380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:06.930358 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:06.930315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:06.931487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:06.931467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:08.694305 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:08.694266    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:06:08.694784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:08.694524    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" podUID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" containerName="istio-proxy" containerID="cri-o://76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448" gracePeriod=30
Sep 16 13:06:11.235763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.235739    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:06:11.314484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314505    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdwbw\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:11.314971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.314881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy\") pod \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\" (UID: \"4b16750c-8fed-4c47-9322-a81eb0c9cf56\") "
Sep 16 13:06:11.315281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.315100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:11.315281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.315194    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-credential-socket\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.315281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.315212    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-socket\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.315281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.315230    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-workload-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.315604 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.315570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:06:11.316047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.316024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:11.317631 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.317582    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:06:11.317723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.317644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw" (OuterVolumeSpecName: "kube-api-access-xdwbw") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "kube-api-access-xdwbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:11.317723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.317660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:11.317848 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.317828    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b16750c-8fed-4c47-9322-a81eb0c9cf56" (UID: "4b16750c-8fed-4c47-9322-a81eb0c9cf56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:11.416012 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.415959    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-token\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.416012 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.416008    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-data\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.416012 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.416018    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwbw\" (UniqueName: \"kubernetes.io/projected/4b16750c-8fed-4c47-9322-a81eb0c9cf56-kube-api-access-xdwbw\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.416240 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.416028    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istiod-ca-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.416240 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.416038    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-podinfo\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.416240 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.416045    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b16750c-8fed-4c47-9322-a81eb0c9cf56-istio-envoy\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:11.958321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.958284    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" containerID="76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448" exitCode=0
Sep 16 13:06:11.958491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.958327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"
Sep 16 13:06:11.958491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.958378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" event={"ID":"4b16750c-8fed-4c47-9322-a81eb0c9cf56","Type":"ContainerDied","Data":"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"}
Sep 16 13:06:11.958491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.958435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd" event={"ID":"4b16750c-8fed-4c47-9322-a81eb0c9cf56","Type":"ContainerDied","Data":"5e0f362669c3d25dd90247b5f3704423d8ce5fc3efe39f688878561d4e9cacee"}
Sep 16 13:06:11.958491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.958464    2579 scope.go:117] "RemoveContainer" containerID="76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"
Sep 16 13:06:11.967337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.967321    2579 scope.go:117] "RemoveContainer" containerID="76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"
Sep 16 13:06:11.967660 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:06:11.967642    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448\": container with ID starting with 76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448 not found: ID does not exist" containerID="76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"
Sep 16 13:06:11.967721 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.967665    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448"} err="failed to get container status \"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448\": rpc error: code = NotFound desc = could not find container \"76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448\": container with ID starting with 76656c42d0c1da6bd5d58ac758873a6c8d3b41d6638b1f093d94447f7df75448 not found: ID does not exist"
Sep 16 13:06:11.979649 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.979624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:06:11.982926 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:11.982906    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-whtbd"]
Sep 16 13:06:12.458831 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:12.458799    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" path="/var/lib/kubelet/pods/4b16750c-8fed-4c47-9322-a81eb0c9cf56/volumes"
Sep 16 13:06:32.200191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:32.200155    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:32.200685 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:32.200409    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerName="istio-proxy" containerID="cri-o://cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a" gracePeriod=30
Sep 16 13:06:35.346664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.346630    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:35.423744 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.423897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423780    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.423897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.423897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.423897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.423897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzr8\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.424157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.424157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.423940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.424157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data\") pod \"5794cc87-2aba-47e0-af29-672729ef1fa0\" (UID: \"5794cc87-2aba-47e0-af29-672729ef1fa0\") "
Sep 16 13:06:35.424157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:35.424341 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data" (OuterVolumeSpecName: "istio-data") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:35.424341 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424306    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.424631 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:06:35.424784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424710    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:35.424862 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.424843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:35.426376 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.426345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8" (OuterVolumeSpecName: "kube-api-access-4wzr8") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "kube-api-access-4wzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:35.426487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.426383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token" (OuterVolumeSpecName: "istio-token") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:35.426487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.426431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:06:35.426790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.426766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5794cc87-2aba-47e0-af29-672729ef1fa0" (UID: "5794cc87-2aba-47e0-af29-672729ef1fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:35.525047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525007    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-credential-socket\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525043    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5794cc87-2aba-47e0-af29-672729ef1fa0-istiod-ca-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525054    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzr8\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-kube-api-access-4wzr8\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525062    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-envoy\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525070    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-token\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525078    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-data\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525086    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5794cc87-2aba-47e0-af29-672729ef1fa0-istio-podinfo\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:35.525291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:35.525093    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5794cc87-2aba-47e0-af29-672729ef1fa0-workload-socket\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:06:36.070782 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.070747    2579 generic.go:356] "Generic (PLEG): container finished" podID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerID="cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a" exitCode=0
Sep 16 13:06:36.070977 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.070803    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"
Sep 16 13:06:36.070977 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.070837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" event={"ID":"5794cc87-2aba-47e0-af29-672729ef1fa0","Type":"ContainerDied","Data":"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"}
Sep 16 13:06:36.070977 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.070895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5" event={"ID":"5794cc87-2aba-47e0-af29-672729ef1fa0","Type":"ContainerDied","Data":"7d3cec32a3be36009bbd28a86229b2075bdf9972ae30100c5c9e5a2bfba45879"}
Sep 16 13:06:36.070977 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.070929    2579 scope.go:117] "RemoveContainer" containerID="cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"
Sep 16 13:06:36.080223 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.080203    2579 scope.go:117] "RemoveContainer" containerID="cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"
Sep 16 13:06:36.080507 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:06:36.080490    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a\": container with ID starting with cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a not found: ID does not exist" containerID="cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"
Sep 16 13:06:36.080583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.080514    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a"} err="failed to get container status \"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a\": rpc error: code = NotFound desc = could not find container \"cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a\": container with ID starting with cf0300327094e9b6cb9e2acf09fe2c13640fb26212de64d82dc81945e9d2797a not found: ID does not exist"
Sep 16 13:06:36.091577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.091533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:36.094957 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.094935    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-n49b5"]
Sep 16 13:06:36.458728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:36.458699    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" path="/var/lib/kubelet/pods/5794cc87-2aba-47e0-af29-672729ef1fa0/volumes"
Sep 16 13:06:46.342006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.341970    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342310    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerName="istio-proxy"
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerName="istio-proxy"
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" containerName="istio-proxy"
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342338    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" containerName="istio-proxy"
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342395    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b16750c-8fed-4c47-9322-a81eb0c9cf56" containerName="istio-proxy"
Sep 16 13:06:46.342485 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.342405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5794cc87-2aba-47e0-af29-672729ef1fa0" containerName="istio-proxy"
Sep 16 13:06:46.344628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.344608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.348448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.348425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 13:06:46.348552 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.348486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-jljl5\""
Sep 16 13:06:46.348860 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.348841    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 13:06:46.348953 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.348889    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 13:06:46.348953 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.348899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-xghsq\""
Sep 16 13:06:46.357325 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.357300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:06:46.423211 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.423184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzds\" (UniqueName: \"kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.423361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.423226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.523907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.523880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzds\" (UniqueName: \"kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.524088 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.523929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.526395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.526372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.532145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.532119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzds\" (UniqueName: \"kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.656345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.656317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:46.931470 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:46.931434    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:06:46.934094 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:06:46.934065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6443c5c6_5b52_47cf_8e7a_173e99075ce6.slice/crio-3a66a52aaa0c046f1b6ef9325ec219c30a29cf01c1d8b84e3bed29a412ac5f96 WatchSource:0}: Error finding container 3a66a52aaa0c046f1b6ef9325ec219c30a29cf01c1d8b84e3bed29a412ac5f96: Status 404 returned error can't find the container with id 3a66a52aaa0c046f1b6ef9325ec219c30a29cf01c1d8b84e3bed29a412ac5f96
Sep 16 13:06:47.125238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:47.125202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" event={"ID":"6443c5c6-5b52-47cf-8e7a-173e99075ce6","Type":"ContainerStarted","Data":"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"}
Sep 16 13:06:47.125469 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:47.125268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" event={"ID":"6443c5c6-5b52-47cf-8e7a-173e99075ce6","Type":"ContainerStarted","Data":"3a66a52aaa0c046f1b6ef9325ec219c30a29cf01c1d8b84e3bed29a412ac5f96"}
Sep 16 13:06:47.125469 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:47.125300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:06:47.148135 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:06:47.148076    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podStartSLOduration=1.148040672 podStartE2EDuration="1.148040672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:47.146864104 +0000 UTC m=+1113.185663227" watchObservedRunningTime="2026-09-16 13:06:47.148040672 +0000 UTC m=+1113.186839797"
Sep 16 13:07:18.131038 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:18.131006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:07:20.336717 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:20.336679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:07:35.358540 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:35.358488    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="main" containerID="cri-o://17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9" gracePeriod=45
Sep 16 13:07:36.006775 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.006752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:07:36.158029 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.157996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lksj\" (UniqueName: \"kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj\") pod \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\" (UID: \"e0271848-6c62-4321-8ae0-ca1ccc1f1e50\") "
Sep 16 13:07:36.158651 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:36.158801 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158776    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:36.158875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.158799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home" (OuterVolumeSpecName: "home") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:36.160784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.160749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj" (OuterVolumeSpecName: "kube-api-access-8lksj") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "kube-api-access-8lksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:07:36.160882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.160796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm" (OuterVolumeSpecName: "dshm") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:36.160882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.160811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:07:36.175531 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.175505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:36.183838 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.183818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0271848-6c62-4321-8ae0-ca1ccc1f1e50" (UID: "e0271848-6c62-4321-8ae0-ca1ccc1f1e50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:36.259071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259048    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259069    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259079    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259088    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259096    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259103    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lksj\" (UniqueName: \"kubernetes.io/projected/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-kube-api-access-8lksj\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259111    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.259185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.259119    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0271848-6c62-4321-8ae0-ca1ccc1f1e50-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:07:36.352886 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.352854    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerID="17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9" exitCode=0
Sep 16 13:07:36.353026 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.352916    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 13:07:36.353026 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.352911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerDied","Data":"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"}
Sep 16 13:07:36.353107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.353037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e0271848-6c62-4321-8ae0-ca1ccc1f1e50","Type":"ContainerDied","Data":"4e7e477f4270de27fea9c36086f4056bd77a07f216f5053845d1ba086bdd4acd"}
Sep 16 13:07:36.353107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.353059    2579 scope.go:117] "RemoveContainer" containerID="17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"
Sep 16 13:07:36.364071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.363732    2579 scope.go:117] "RemoveContainer" containerID="6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"
Sep 16 13:07:36.371605 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.371589    2579 scope.go:117] "RemoveContainer" containerID="17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"
Sep 16 13:07:36.371862 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:07:36.371845    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9\": container with ID starting with 17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9 not found: ID does not exist" containerID="17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"
Sep 16 13:07:36.371900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.371871    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9"} err="failed to get container status \"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9\": rpc error: code = NotFound desc = could not find container \"17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9\": container with ID starting with 17661ea249c040222ea7c5374eb6c54ec5ca7e8651388c8e2f80d29c04fce7e9 not found: ID does not exist"
Sep 16 13:07:36.371900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.371888    2579 scope.go:117] "RemoveContainer" containerID="6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"
Sep 16 13:07:36.372145 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:07:36.372128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f\": container with ID starting with 6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f not found: ID does not exist" containerID="6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"
Sep 16 13:07:36.372188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.372149    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f"} err="failed to get container status \"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f\": rpc error: code = NotFound desc = could not find container \"6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f\": container with ID starting with 6ec6d6991a387d07dbc0cc18b10e532180db7de48c083f8183dd27d3b5a0199f not found: ID does not exist"
Sep 16 13:07:36.377338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.377298    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:07:36.381465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.381444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 13:07:36.462442 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:07:36.462377    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" path="/var/lib/kubelet/pods/e0271848-6c62-4321-8ae0-ca1ccc1f1e50/volumes"
Sep 16 13:08:07.280811 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.280779    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:07.281361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.281106    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="storage-initializer"
Sep 16 13:08:07.281361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.281123    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="storage-initializer"
Sep 16 13:08:07.281361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.281135    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="main"
Sep 16 13:08:07.281361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.281144    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="main"
Sep 16 13:08:07.281361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.281260    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0271848-6c62-4321-8ae0-ca1ccc1f1e50" containerName="main"
Sep 16 13:08:07.283853 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.283834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.287652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.287631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 13:08:07.287652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.287644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 13:08:07.287807 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.287655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-fz5lj\""
Sep 16 13:08:07.287807 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.287709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 13:08:07.292648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.292628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:07.298058 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.298146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.298146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.298146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.298250 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4kdg\" (UniqueName: \"kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.298284 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.298243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4kdg\" (UniqueName: \"kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399873 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.399967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.399917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.401786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.401763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.401976 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.401960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.407949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.407916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4kdg\" (UniqueName: \"kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg\") pod \"scheduler-inline-config-test-kserve-8655bc757-k9skx\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.594275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.594189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:07.866491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:07.866387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:07.869099 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:08:07.869072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7836865c_c793_49e4_b7ed_9a278f6643a8.slice/crio-25e26617ba0111ec3b844c0fa0f7a603796ab4121c9a21de26a25da72463fb7f WatchSource:0}: Error finding container 25e26617ba0111ec3b844c0fa0f7a603796ab4121c9a21de26a25da72463fb7f: Status 404 returned error can't find the container with id 25e26617ba0111ec3b844c0fa0f7a603796ab4121c9a21de26a25da72463fb7f
Sep 16 13:08:08.484347 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:08.484314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" event={"ID":"7836865c-c793-49e4-b7ed-9a278f6643a8","Type":"ContainerStarted","Data":"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"}
Sep 16 13:08:08.484758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:08.484359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" event={"ID":"7836865c-c793-49e4-b7ed-9a278f6643a8","Type":"ContainerStarted","Data":"25e26617ba0111ec3b844c0fa0f7a603796ab4121c9a21de26a25da72463fb7f"}
Sep 16 13:08:08.521320 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:08.521266    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" podStartSLOduration=1.521247154 podStartE2EDuration="1.521247154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:08.519997656 +0000 UTC m=+1194.558796779" watchObservedRunningTime="2026-09-16 13:08:08.521247154 +0000 UTC m=+1194.560046276"
Sep 16 13:08:08.595060 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:08.595017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:14.407211 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:14.407174    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:08:14.407635 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:14.407481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:08:17.594656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:17.594616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:17.606856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:17.606825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:18.535659 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:18.535629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:40.541882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.541846    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:40.550763 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:08:40.550724    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:08:40.550763 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:08:40.550763 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:08:40.550763 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280" cmd=["/bin/sleep","15"]
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.550771    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" message="" err=<
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:08:40.550783    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerName="main" containerID="cri-o://f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"
Sep 16 13:08:40.551048 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.550825    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerName="main" containerID="cri-o://f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280" gracePeriod=60
Sep 16 13:08:40.892921 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.892900    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:40.981752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.981927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.981927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.981927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4kdg\" (UniqueName: \"kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.981927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.981927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.981911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache\") pod \"7836865c-c793-49e4-b7ed-9a278f6643a8\" (UID: \"7836865c-c793-49e4-b7ed-9a278f6643a8\") "
Sep 16 13:08:40.982183 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.982037    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:08:40.982183 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.982119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home" (OuterVolumeSpecName: "home") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:08:40.982306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.982190    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:40.982306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.982208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:08:40.984154 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.984127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm" (OuterVolumeSpecName: "dshm") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:08:40.984260 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.984198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg" (OuterVolumeSpecName: "kube-api-access-g4kdg") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "kube-api-access-g4kdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:40.984260 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:40.984205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7836865c-c793-49e4-b7ed-9a278f6643a8" (UID: "7836865c-c793-49e4-b7ed-9a278f6643a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:41.082919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.082799    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4kdg\" (UniqueName: \"kubernetes.io/projected/7836865c-c793-49e4-b7ed-9a278f6643a8-kube-api-access-g4kdg\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:41.082919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.082828    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:41.082919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.082840    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:41.082919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.082848    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7836865c-c793-49e4-b7ed-9a278f6643a8-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:41.082919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.082855    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7836865c-c793-49e4-b7ed-9a278f6643a8-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:08:41.624895 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.624860    2579 generic.go:356] "Generic (PLEG): container finished" podID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerID="f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280" exitCode=0
Sep 16 13:08:41.624895 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.624896    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"
Sep 16 13:08:41.625362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.624933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" event={"ID":"7836865c-c793-49e4-b7ed-9a278f6643a8","Type":"ContainerDied","Data":"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"}
Sep 16 13:08:41.625362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.624976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx" event={"ID":"7836865c-c793-49e4-b7ed-9a278f6643a8","Type":"ContainerDied","Data":"25e26617ba0111ec3b844c0fa0f7a603796ab4121c9a21de26a25da72463fb7f"}
Sep 16 13:08:41.625362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.625001    2579 scope.go:117] "RemoveContainer" containerID="f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"
Sep 16 13:08:41.634253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.634228    2579 scope.go:117] "RemoveContainer" containerID="f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"
Sep 16 13:08:41.634505 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:08:41.634486    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280\": container with ID starting with f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280 not found: ID does not exist" containerID="f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"
Sep 16 13:08:41.634547 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.634515    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280"} err="failed to get container status \"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280\": rpc error: code = NotFound desc = could not find container \"f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280\": container with ID starting with f4e3339661215d21751c1a7bef5d1d0fd02399b8b205d091331c123c0c4f8280 not found: ID does not exist"
Sep 16 13:08:41.646322 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.646294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:41.648165 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:41.648141    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-k9skx"]
Sep 16 13:08:42.457247 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:42.457215    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" path="/var/lib/kubelet/pods/7836865c-c793-49e4-b7ed-9a278f6643a8/volumes"
Sep 16 13:08:51.411772 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:51.411734    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:08:58.130808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:08:58.130770    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:06.431640 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:06.431602    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main" containerID="cri-o://5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257" gracePeriod=45
Sep 16 13:09:08.130747 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:08.130704    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:18.131331 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:18.131292    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:18.131725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:18.131391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:09:28.131283 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:28.131243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:36.695187 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.695106    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:09:36.801683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.801647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkzds\" (UniqueName: \"kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds\") pod \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") "
Sep 16 13:09:36.801683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.801690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs\") pod \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\" (UID: \"6443c5c6-5b52-47cf-8e7a-173e99075ce6\") "
Sep 16 13:09:36.804055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.804016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6443c5c6-5b52-47cf-8e7a-173e99075ce6" (UID: "6443c5c6-5b52-47cf-8e7a-173e99075ce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:36.804190 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.804077    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds" (OuterVolumeSpecName: "kube-api-access-tkzds") pod "6443c5c6-5b52-47cf-8e7a-173e99075ce6" (UID: "6443c5c6-5b52-47cf-8e7a-173e99075ce6"). InnerVolumeSpecName "kube-api-access-tkzds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:36.855109 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.855080    2579 generic.go:356] "Generic (PLEG): container finished" podID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerID="5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257" exitCode=0
Sep 16 13:09:36.855327 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.855126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"
Sep 16 13:09:36.855327 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.855175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" event={"ID":"6443c5c6-5b52-47cf-8e7a-173e99075ce6","Type":"ContainerDied","Data":"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"}
Sep 16 13:09:36.855327 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.855229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl" event={"ID":"6443c5c6-5b52-47cf-8e7a-173e99075ce6","Type":"ContainerDied","Data":"3a66a52aaa0c046f1b6ef9325ec219c30a29cf01c1d8b84e3bed29a412ac5f96"}
Sep 16 13:09:36.855327 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.855253    2579 scope.go:117] "RemoveContainer" containerID="5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"
Sep 16 13:09:36.863922 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.863898    2579 scope.go:117] "RemoveContainer" containerID="5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"
Sep 16 13:09:36.864307 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:09:36.864284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257\": container with ID starting with 5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257 not found: ID does not exist" containerID="5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"
Sep 16 13:09:36.864357 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.864318    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257"} err="failed to get container status \"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257\": rpc error: code = NotFound desc = could not find container \"5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257\": container with ID starting with 5dc74e45a2baf25080199bf3e7a812e077014ec9af39afb0cbfd1ee3b2060257 not found: ID does not exist"
Sep 16 13:09:36.876180 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.876148    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:09:36.879379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.879349    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6fkhl"]
Sep 16 13:09:36.903061 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.903026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzds\" (UniqueName: \"kubernetes.io/projected/6443c5c6-5b52-47cf-8e7a-173e99075ce6-kube-api-access-tkzds\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:09:36.903061 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.903060    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6443c5c6-5b52-47cf-8e7a-173e99075ce6-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:09:36.970536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.970456    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:09:36.970883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.970864    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main"
Sep 16 13:09:36.970934 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.970887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main"
Sep 16 13:09:36.970934 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.970915    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerName="main"
Sep 16 13:09:36.970934 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.970923    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerName="main"
Sep 16 13:09:36.971209 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.971007    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7836865c-c793-49e4-b7ed-9a278f6643a8" containerName="main"
Sep 16 13:09:36.971209 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.971018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" containerName="main"
Sep 16 13:09:36.975230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.975203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:36.977938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.977907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 13:09:36.977938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.977917    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 13:09:36.978141 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.977972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-w6d9w\""
Sep 16 13:09:36.978141 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.977994    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 13:09:36.978141 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.978094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-hngvz\""
Sep 16 13:09:36.980656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:36.980634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:09:37.004408 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.004373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lsf\" (UniqueName: \"kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.004648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.004433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.104954 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.104913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.105144 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.105009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lsf\" (UniqueName: \"kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.107466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.107435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.112999 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.112975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lsf\" (UniqueName: \"kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.286862 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.286688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.565334 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.564932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:09:37.568162 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:09:37.568133    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1cfcef_7db7_478b_91a9_e0d055db8ff3.slice/crio-ee41f425aee8274e5d59443d8c0966c5038b991d456d2f60517afed7dd17cb4a WatchSource:0}: Error finding container ee41f425aee8274e5d59443d8c0966c5038b991d456d2f60517afed7dd17cb4a: Status 404 returned error can't find the container with id ee41f425aee8274e5d59443d8c0966c5038b991d456d2f60517afed7dd17cb4a
Sep 16 13:09:37.865610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.865473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" event={"ID":"2f1cfcef-7db7-478b-91a9-e0d055db8ff3","Type":"ContainerStarted","Data":"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"}
Sep 16 13:09:37.865610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.865585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" event={"ID":"2f1cfcef-7db7-478b-91a9-e0d055db8ff3","Type":"ContainerStarted","Data":"ee41f425aee8274e5d59443d8c0966c5038b991d456d2f60517afed7dd17cb4a"}
Sep 16 13:09:37.866625 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.866597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:09:37.891515 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:37.891457    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podStartSLOduration=1.891438892 podStartE2EDuration="1.891438892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:37.889881768 +0000 UTC m=+1283.928680895" watchObservedRunningTime="2026-09-16 13:09:37.891438892 +0000 UTC m=+1283.930238015"
Sep 16 13:09:38.460964 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:38.460914    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6443c5c6-5b52-47cf-8e7a-173e99075ce6" path="/var/lib/kubelet/pods/6443c5c6-5b52-47cf-8e7a-173e99075ce6/volumes"
Sep 16 13:09:51.695274 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.695236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:09:51.698813 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.698790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.701500 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 13:09:51.701641 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 13:09:51.701641 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8cf6p\""
Sep 16 13:09:51.701757 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701672    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 13:09:51.701816 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:09:51.701923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.701905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 13:09:51.710080 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.710055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:09:51.816856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.816827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.816862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9924\" (UniqueName: \"kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.816918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.816962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.816985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.817019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.817063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.817204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.817093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917731 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.917875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.917959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9924\" (UniqueName: \"kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.918119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.918315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.918324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918613 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.918592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.918763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.918743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.920243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.920218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.920328 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.920296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:51.925070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:51.925050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9924\" (UniqueName: \"kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924\") pod \"auth-invalid-token-test-kserve-6d4d995dd4-hn842\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:52.010355 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:52.010298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:09:52.298623 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:52.294207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:09:52.299985 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:09:52.299942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6707d78a_168c_4f2d_95a8_26cbdeeafd07.slice/crio-17a742d9c0dde0ecd7d8ec08b12fa6934cf2ca35969f20a533aff1385de41bfa WatchSource:0}: Error finding container 17a742d9c0dde0ecd7d8ec08b12fa6934cf2ca35969f20a533aff1385de41bfa: Status 404 returned error can't find the container with id 17a742d9c0dde0ecd7d8ec08b12fa6934cf2ca35969f20a533aff1385de41bfa
Sep 16 13:09:52.942868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:52.942833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerStarted","Data":"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"}
Sep 16 13:09:52.943298 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:52.942888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerStarted","Data":"17a742d9c0dde0ecd7d8ec08b12fa6934cf2ca35969f20a533aff1385de41bfa"}
Sep 16 13:09:54.953676 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:54.953647    2579 generic.go:356] "Generic (PLEG): container finished" podID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerID="a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce" exitCode=0
Sep 16 13:09:54.954050 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:54.953687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerDied","Data":"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"}
Sep 16 13:09:55.959764 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:55.959732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerStarted","Data":"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"}
Sep 16 13:09:55.983770 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:55.983717    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podStartSLOduration=2.724648023 podStartE2EDuration="4.983700838s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.259052815s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:55.981438433 +0000 UTC m=+1302.020237557" watchObservedRunningTime="2026-09-16 13:09:55.983700838 +0000 UTC m=+1302.022499963"
Sep 16 13:09:56.011229 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:09:56.011194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:10:02.010756 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:02.010714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:10:02.012004 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:02.011969    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:09.886199 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:09.886167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:10:11.227871 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:11.226941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:10:12.011009 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:12.010966    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:19.886250 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:19.886209    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:10:22.011679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:22.011641    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:26.256843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:26.256778    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main" containerID="cri-o://8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f" gracePeriod=45
Sep 16 13:10:29.886230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:29.886193    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:10:32.011405 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:32.011367    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:39.887375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:39.887284    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:10:39.887894 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:39.887390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:10:42.010719 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:42.010679    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:49.887187 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:49.887147    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:10:52.011781 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:52.011736    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:10:56.498447 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.498421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:10:56.580664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.580576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4lsf\" (UniqueName: \"kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf\") pod \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") "
Sep 16 13:10:56.580808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.580671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs\") pod \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\" (UID: \"2f1cfcef-7db7-478b-91a9-e0d055db8ff3\") "
Sep 16 13:10:56.582951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.582918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf" (OuterVolumeSpecName: "kube-api-access-l4lsf") pod "2f1cfcef-7db7-478b-91a9-e0d055db8ff3" (UID: "2f1cfcef-7db7-478b-91a9-e0d055db8ff3"). InnerVolumeSpecName "kube-api-access-l4lsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:56.583107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.583014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f1cfcef-7db7-478b-91a9-e0d055db8ff3" (UID: "2f1cfcef-7db7-478b-91a9-e0d055db8ff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:10:56.681868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.681817    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lsf\" (UniqueName: \"kubernetes.io/projected/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-kube-api-access-l4lsf\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:10:56.681868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:56.681853    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1cfcef-7db7-478b-91a9-e0d055db8ff3-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:10:57.221457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.221420    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerID="8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f" exitCode=0
Sep 16 13:10:57.221679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.221465    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"
Sep 16 13:10:57.221679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.221476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" event={"ID":"2f1cfcef-7db7-478b-91a9-e0d055db8ff3","Type":"ContainerDied","Data":"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"}
Sep 16 13:10:57.221679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.221520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2" event={"ID":"2f1cfcef-7db7-478b-91a9-e0d055db8ff3","Type":"ContainerDied","Data":"ee41f425aee8274e5d59443d8c0966c5038b991d456d2f60517afed7dd17cb4a"}
Sep 16 13:10:57.221679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.221541    2579 scope.go:117] "RemoveContainer" containerID="8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"
Sep 16 13:10:57.230446 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.230426    2579 scope.go:117] "RemoveContainer" containerID="8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"
Sep 16 13:10:57.230741 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:10:57.230724    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f\": container with ID starting with 8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f not found: ID does not exist" containerID="8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"
Sep 16 13:10:57.230803 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.230748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f"} err="failed to get container status \"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f\": rpc error: code = NotFound desc = could not find container \"8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f\": container with ID starting with 8ad10bba0a3de59211a5cb428d78e0449e6add3224dce174e8583ef8773a6c7f not found: ID does not exist"
Sep 16 13:10:57.242917 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.242893    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:10:57.248883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:57.248863    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetjzw2"]
Sep 16 13:10:58.457487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:10:58.457453    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" path="/var/lib/kubelet/pods/2f1cfcef-7db7-478b-91a9-e0d055db8ff3/volumes"
Sep 16 13:11:02.010932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:02.010892    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:11:12.011009 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:12.010957    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:11:20.479525 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.479493    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:11:20.479954 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.479801    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main"
Sep 16 13:11:20.479954 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.479814    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main"
Sep 16 13:11:20.479954 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.479867    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1cfcef-7db7-478b-91a9-e0d055db8ff3" containerName="main"
Sep 16 13:11:20.482573 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.482542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.485427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.485407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 13:11:20.486848 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.486822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 13:11:20.486938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.486850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-4mqn6\""
Sep 16 13:11:20.486938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.486847    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 13:11:20.487164 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.487146    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-n9fkv\""
Sep 16 13:11:20.494441 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.494422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:11:20.592888 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.592856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.593060 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.592936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgp4\" (UniqueName: \"kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.693801 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.693751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgp4\" (UniqueName: \"kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.693990 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.693840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.696329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.696297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.702333 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.702305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgp4\" (UniqueName: \"kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:20.795672 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:20.795552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:21.066734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:21.066636    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:11:21.072319 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:11:21.072292    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a67222c_9495_4ad2_bba2_6d5d3cd0e9ed.slice/crio-f0f6edeefb0bfd61e36a4d8a1a7220af6c07be2fbbb57f0789349fb6010bcf4b WatchSource:0}: Error finding container f0f6edeefb0bfd61e36a4d8a1a7220af6c07be2fbbb57f0789349fb6010bcf4b: Status 404 returned error can't find the container with id f0f6edeefb0bfd61e36a4d8a1a7220af6c07be2fbbb57f0789349fb6010bcf4b
Sep 16 13:11:21.330874 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:21.330782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" event={"ID":"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed","Type":"ContainerStarted","Data":"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"}
Sep 16 13:11:21.330874 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:21.330837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" event={"ID":"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed","Type":"ContainerStarted","Data":"f0f6edeefb0bfd61e36a4d8a1a7220af6c07be2fbbb57f0789349fb6010bcf4b"}
Sep 16 13:11:21.331060 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:21.330909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:21.348891 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:21.348832    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podStartSLOduration=1.348811566 podStartE2EDuration="1.348811566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:21.346620386 +0000 UTC m=+1387.385419508" watchObservedRunningTime="2026-09-16 13:11:21.348811566 +0000 UTC m=+1387.387610690"
Sep 16 13:11:22.011365 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:22.011323    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:11:32.010832 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:32.010793    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:11:42.011205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:42.011150    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Sep 16 13:11:52.021025 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:52.020989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:11:52.028944 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:52.028918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:11:52.338379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:52.338292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:11:54.169049 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:54.169007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:11:55.124749 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:11:55.124717    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:12:02.337660 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:02.337621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:09.188425 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:09.188333    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main" containerID="cri-o://2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e" gracePeriod=45
Sep 16 13:12:10.146906 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.146859    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main" containerID="cri-o://f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df" gracePeriod=45
Sep 16 13:12:10.894449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.894425    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:12:10.927137 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927137 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9924\" (UniqueName: \"kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home\") pod \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\" (UID: \"6707d78a-168c-4f2d-95a8-26cbdeeafd07\") "
Sep 16 13:12:10.927691 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache" (OuterVolumeSpecName: "model-cache") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:10.927691 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.927662    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:10.928222 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.928181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home" (OuterVolumeSpecName: "home") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:10.928222 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.928192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:12:10.930038 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.929996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:10.930197 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.930169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm" (OuterVolumeSpecName: "dshm") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:10.930336 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.930314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924" (OuterVolumeSpecName: "kube-api-access-r9924") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "kube-api-access-r9924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:10.940145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.940099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:10.950342 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:10.950319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6707d78a-168c-4f2d-95a8-26cbdeeafd07" (UID: "6707d78a-168c-4f2d-95a8-26cbdeeafd07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:11.028337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028302    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028332    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028346    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028357    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9924\" (UniqueName: \"kubernetes.io/projected/6707d78a-168c-4f2d-95a8-26cbdeeafd07-kube-api-access-r9924\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028371    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6707d78a-168c-4f2d-95a8-26cbdeeafd07-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028384    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d78a-168c-4f2d-95a8-26cbdeeafd07-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.028554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.028393    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d78a-168c-4f2d-95a8-26cbdeeafd07-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:11.536258 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.536225    2579 generic.go:356] "Generic (PLEG): container finished" podID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerID="f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df" exitCode=0
Sep 16 13:12:11.536432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.536267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerDied","Data":"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"}
Sep 16 13:12:11.536432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.536278    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"
Sep 16 13:12:11.536432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.536309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842" event={"ID":"6707d78a-168c-4f2d-95a8-26cbdeeafd07","Type":"ContainerDied","Data":"17a742d9c0dde0ecd7d8ec08b12fa6934cf2ca35969f20a533aff1385de41bfa"}
Sep 16 13:12:11.536432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.536333    2579 scope.go:117] "RemoveContainer" containerID="f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"
Sep 16 13:12:11.545390 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.545367    2579 scope.go:117] "RemoveContainer" containerID="a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"
Sep 16 13:12:11.552737 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.552708    2579 scope.go:117] "RemoveContainer" containerID="f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"
Sep 16 13:12:11.552989 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:12:11.552967    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df\": container with ID starting with f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df not found: ID does not exist" containerID="f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"
Sep 16 13:12:11.553063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.553001    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df"} err="failed to get container status \"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df\": rpc error: code = NotFound desc = could not find container \"f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df\": container with ID starting with f8a0c2565cfe257fdb227e85ed52de85a2f71ddc59240c89e61cc269f27069df not found: ID does not exist"
Sep 16 13:12:11.553063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.553026    2579 scope.go:117] "RemoveContainer" containerID="a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"
Sep 16 13:12:11.553293 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:12:11.553274    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce\": container with ID starting with a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce not found: ID does not exist" containerID="a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"
Sep 16 13:12:11.553345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.553297    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce"} err="failed to get container status \"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce\": rpc error: code = NotFound desc = could not find container \"a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce\": container with ID starting with a3f157494447319dfd5526823b450295746f72cc77c2908e4f7c6ba47d5662ce not found: ID does not exist"
Sep 16 13:12:11.559127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.559100    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:12:11.564606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:11.564584    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6d4d995dd4-hn842"]
Sep 16 13:12:12.337906 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:12.337867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:12.458293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:12.458260    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" path="/var/lib/kubelet/pods/6707d78a-168c-4f2d-95a8-26cbdeeafd07/volumes"
Sep 16 13:12:22.338085 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:22.338043    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:22.338532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:22.338178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:12:32.338238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:32.338197    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:39.432888 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.432856    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:12:39.557583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.557462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shgp4\" (UniqueName: \"kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4\") pod \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") "
Sep 16 13:12:39.557583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.557526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs\") pod \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\" (UID: \"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed\") "
Sep 16 13:12:39.559746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.559719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4" (OuterVolumeSpecName: "kube-api-access-shgp4") pod "8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" (UID: "8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed"). InnerVolumeSpecName "kube-api-access-shgp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:39.559842 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.559738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" (UID: "8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:39.654073 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.654035    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerID="2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e" exitCode=0
Sep 16 13:12:39.654239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.654087    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"
Sep 16 13:12:39.654239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.654089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" event={"ID":"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed","Type":"ContainerDied","Data":"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"}
Sep 16 13:12:39.654239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.654144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh" event={"ID":"8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed","Type":"ContainerDied","Data":"f0f6edeefb0bfd61e36a4d8a1a7220af6c07be2fbbb57f0789349fb6010bcf4b"}
Sep 16 13:12:39.654239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.654173    2579 scope.go:117] "RemoveContainer" containerID="2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"
Sep 16 13:12:39.658176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.658149    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgp4\" (UniqueName: \"kubernetes.io/projected/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-kube-api-access-shgp4\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:39.658176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.658174    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:12:39.662421 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.662401    2579 scope.go:117] "RemoveContainer" containerID="2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"
Sep 16 13:12:39.662760 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:12:39.662740    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e\": container with ID starting with 2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e not found: ID does not exist" containerID="2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"
Sep 16 13:12:39.662837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.662766    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e"} err="failed to get container status \"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e\": rpc error: code = NotFound desc = could not find container \"2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e\": container with ID starting with 2b4d251397f2d6d942b231f828a017a91e174faff284a73fc477cf3aec949e3e not found: ID does not exist"
Sep 16 13:12:39.676065 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.676034    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:12:39.677739 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:39.677718    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bsxjh"]
Sep 16 13:12:40.457967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:12:40.457933    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" path="/var/lib/kubelet/pods/8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed/volumes"
Sep 16 13:13:06.671885 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.671851    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672166    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672183    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672204    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="storage-initializer"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="storage-initializer"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672227    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672233    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672307    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707d78a-168c-4f2d-95a8-26cbdeeafd07" containerName="main"
Sep 16 13:13:06.672465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.672317    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a67222c-9495-4ad2-bba2-6d5d3cd0e9ed" containerName="main"
Sep 16 13:13:06.674584 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.674554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.677256 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.677209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 13:13:06.678459 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.678433    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-cvqls\""
Sep 16 13:13:06.678613 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.678489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 13:13:06.678613 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.678592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-28tjj\""
Sep 16 13:13:06.678728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.678626    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 13:13:06.684203 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.684180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:13:06.789705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.789673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.789898 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.789720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbw6t\" (UniqueName: \"kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.890532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.890492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.890726 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.890543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw6t\" (UniqueName: \"kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.893115 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.893084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.898379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.898353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbw6t\" (UniqueName: \"kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:06.985167 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:06.985069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:07.264368 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:07.264264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:13:07.781353 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:07.781303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" event={"ID":"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d","Type":"ContainerStarted","Data":"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"}
Sep 16 13:13:07.781827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:07.781373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" event={"ID":"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d","Type":"ContainerStarted","Data":"f7324cb84379748685d00a53d4cbb44fd9200640fe5f0ec61dc33cfa91749e55"}
Sep 16 13:13:07.781827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:07.781427    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:13:07.799710 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:07.799659    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podStartSLOduration=1.7996453 podStartE2EDuration="1.7996453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:07.798104894 +0000 UTC m=+1493.836904022" watchObservedRunningTime="2026-09-16 13:13:07.7996453 +0000 UTC m=+1493.838444423"
Sep 16 13:13:14.430756 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:14.430729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:13:14.432167 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:14.432141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:13:30.005178 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.005140    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:13:30.008076 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.008048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.010665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.010641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 13:13:30.010809 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.010708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:13:30.010809 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.010746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 13:13:30.010809 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.010763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 13:13:30.010991 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.010899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-dn8fc\""
Sep 16 13:13:30.011074 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.011046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 13:13:30.021813 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.021787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:13:30.081708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.081928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ph9\" (UniqueName: \"kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.082097 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.081960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ph9\" (UniqueName: \"kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183998 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183998 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.183998 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.183992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.184175 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.184040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.184293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.184272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.185799 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.185784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.186057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.186039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.191648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.191626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ph9\" (UniqueName: \"kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9\") pod \"auth-disabled-test-kserve-78bcf54b99-dfmrk\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.292184 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.289956    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:13:30.295199 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.293983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.296899 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.296876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-hdm9f\""
Sep 16 13:13:30.302122 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.302096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:13:30.319775 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.319748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:30.385319 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.385280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.385488 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.385434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w4m\" (UniqueName: \"kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.486177 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.486137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w4m\" (UniqueName: \"kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.486362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.486198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.488776 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.488753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.494318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.494290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w4m\" (UniqueName: \"kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.593626 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.593444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:13:30.596310 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:13:30.596282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49d8f8b_5546_4d35_b0ec_e97dab388471.slice/crio-54edac034f4acfe1ddac12348144d6f18073218ec1908346c92c0abcefb2bb82 WatchSource:0}: Error finding container 54edac034f4acfe1ddac12348144d6f18073218ec1908346c92c0abcefb2bb82: Status 404 returned error can't find the container with id 54edac034f4acfe1ddac12348144d6f18073218ec1908346c92c0abcefb2bb82
Sep 16 13:13:30.607765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.607744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:30.877625 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.877504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerStarted","Data":"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"}
Sep 16 13:13:30.877625 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.877585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerStarted","Data":"54edac034f4acfe1ddac12348144d6f18073218ec1908346c92c0abcefb2bb82"}
Sep 16 13:13:30.887189 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:30.887158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:13:31.884176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:31.884129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" event={"ID":"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d","Type":"ContainerStarted","Data":"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"}
Sep 16 13:13:31.884671 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:31.884190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" event={"ID":"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d","Type":"ContainerStarted","Data":"c866022a523149c3328c23373b50cb1239369dadd282bdcff3ee6b8c7f560780"}
Sep 16 13:13:31.902330 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:31.902284    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podStartSLOduration=1.902270534 podStartE2EDuration="1.902270534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:31.900929718 +0000 UTC m=+1517.939728841" watchObservedRunningTime="2026-09-16 13:13:31.902270534 +0000 UTC m=+1517.941069656"
Sep 16 13:13:32.888628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:32.888591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:13:33.894022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:33.893990    2579 generic.go:356] "Generic (PLEG): container finished" podID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerID="bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0" exitCode=0
Sep 16 13:13:33.894414 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:33.894063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerDied","Data":"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"}
Sep 16 13:13:34.901018 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:34.900983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerStarted","Data":"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"}
Sep 16 13:13:34.929186 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:34.929124    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podStartSLOduration=3.572166279 podStartE2EDuration="5.929102797s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.356936518s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:34.926874922 +0000 UTC m=+1520.965674046" watchObservedRunningTime="2026-09-16 13:13:34.929102797 +0000 UTC m=+1520.967901921"
Sep 16 13:13:35.320214 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:35.320118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:38.786584 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:38.786477    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:13:40.320353 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:40.320314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:13:40.322195 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:40.322162    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:13:48.787542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:48.787500    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:13:50.321325 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:50.321274    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:13:54.915842 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:54.915798    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:13:54.920860 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:13:54.920822    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:00.320382 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:00.320339    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:14:03.896386 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:03.896358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:14:04.917743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:04.917703    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:09.939337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:09.939292    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" containerID="cri-o://a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39" gracePeriod=45
Sep 16 13:14:10.320717 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:10.320619    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:14:14.918176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:14.918129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:20.320832 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:20.320784    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:14:24.917907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:24.917870    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:30.320973 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:30.320924    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:14:34.918599 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:34.918539    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:40.187160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.187136    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:14:40.195483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.195454    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerID="a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39" exitCode=0
Sep 16 13:14:40.195642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.195494    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"
Sep 16 13:14:40.195642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.195501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" event={"ID":"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d","Type":"ContainerDied","Data":"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"}
Sep 16 13:14:40.195642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.195548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz" event={"ID":"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d","Type":"ContainerDied","Data":"f7324cb84379748685d00a53d4cbb44fd9200640fe5f0ec61dc33cfa91749e55"}
Sep 16 13:14:40.195642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.195608    2579 scope.go:117] "RemoveContainer" containerID="a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"
Sep 16 13:14:40.203712 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.203686    2579 scope.go:117] "RemoveContainer" containerID="a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"
Sep 16 13:14:40.204199 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:14:40.204162    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39\": container with ID starting with a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39 not found: ID does not exist" containerID="a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"
Sep 16 13:14:40.204318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.204209    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39"} err="failed to get container status \"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39\": rpc error: code = NotFound desc = could not find container \"a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39\": container with ID starting with a1a188498fc0645818cd1d7802b9e3d4c2d9e2c24775f8c994250540f9e5fb39 not found: ID does not exist"
Sep 16 13:14:40.305865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.305827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs\") pod \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") "
Sep 16 13:14:40.305865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.305875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbw6t\" (UniqueName: \"kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t\") pod \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\" (UID: \"2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d\") "
Sep 16 13:14:40.308151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.308112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t" (OuterVolumeSpecName: "kube-api-access-nbw6t") pod "2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" (UID: "2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d"). InnerVolumeSpecName "kube-api-access-nbw6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:40.308268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.308124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" (UID: "2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:40.320612 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.320580    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:14:40.406385 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.406344    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbw6t\" (UniqueName: \"kubernetes.io/projected/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-kube-api-access-nbw6t\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:14:40.406545 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.406401    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:14:40.510706 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.510675    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:14:40.514162 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:40.514133    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ltlrz"]
Sep 16 13:14:42.457960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:42.457921    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" path="/var/lib/kubelet/pods/2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d/volumes"
Sep 16 13:14:50.320733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:14:50.320687    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:15:00.320446 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:00.320392    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:15:10.321130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:10.321014    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:15:20.320511 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:20.320462    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:15:30.330157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:30.330125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:15:30.338608 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:30.338550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:15:34.928365 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:34.928333    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:15:34.935499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:34.935461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:15:43.896144 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:43.896104    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:15:49.952050 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:49.951976    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" containerID="cri-o://7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01" gracePeriod=45
Sep 16 13:15:49.954866 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:49.954818    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main" containerID="cri-o://20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0" gracePeriod=45
Sep 16 13:15:50.331832 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.331707    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 13:15:50.694313 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.694283    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:15:50.702672 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6ph9\" (UniqueName: \"kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.702796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.702796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.702796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.702796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.703014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.703014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.703014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.702927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home\") pod \"d49d8f8b-5546-4d35-b0ec-e97dab388471\" (UID: \"d49d8f8b-5546-4d35-b0ec-e97dab388471\") "
Sep 16 13:15:50.703158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.703045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache" (OuterVolumeSpecName: "model-cache") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:50.703158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.703137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:50.703274 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.703162    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.703653 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.703633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home" (OuterVolumeSpecName: "home") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:50.705354 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.705323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:50.705354 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.705341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9" (OuterVolumeSpecName: "kube-api-access-r6ph9") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "kube-api-access-r6ph9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:50.705489 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.705367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm" (OuterVolumeSpecName: "dshm") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:50.721628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.721600    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:50.728134 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.728108    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d49d8f8b-5546-4d35-b0ec-e97dab388471" (UID: "d49d8f8b-5546-4d35-b0ec-e97dab388471"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:50.804412 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804372    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804412 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804404    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804412 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804415    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804692 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804424    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ph9\" (UniqueName: \"kubernetes.io/projected/d49d8f8b-5546-4d35-b0ec-e97dab388471-kube-api-access-r6ph9\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804692 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804435    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49d8f8b-5546-4d35-b0ec-e97dab388471-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804692 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804444    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49d8f8b-5546-4d35-b0ec-e97dab388471-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:50.804692 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:50.804451    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49d8f8b-5546-4d35-b0ec-e97dab388471-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:15:51.508407 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.508347    2579 generic.go:356] "Generic (PLEG): container finished" podID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerID="7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01" exitCode=0
Sep 16 13:15:51.508935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.508424    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"
Sep 16 13:15:51.508935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.508427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerDied","Data":"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"}
Sep 16 13:15:51.508935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.508485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk" event={"ID":"d49d8f8b-5546-4d35-b0ec-e97dab388471","Type":"ContainerDied","Data":"54edac034f4acfe1ddac12348144d6f18073218ec1908346c92c0abcefb2bb82"}
Sep 16 13:15:51.508935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.508507    2579 scope.go:117] "RemoveContainer" containerID="7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"
Sep 16 13:15:51.516614 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.516592    2579 scope.go:117] "RemoveContainer" containerID="bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"
Sep 16 13:15:51.524350 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.524334    2579 scope.go:117] "RemoveContainer" containerID="7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"
Sep 16 13:15:51.524657 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:15:51.524635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01\": container with ID starting with 7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01 not found: ID does not exist" containerID="7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"
Sep 16 13:15:51.524711 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.524667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01"} err="failed to get container status \"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01\": rpc error: code = NotFound desc = could not find container \"7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01\": container with ID starting with 7fee8d7e94248516c12191f69751124cbefcadc217db5e12bf862e5b73ccfe01 not found: ID does not exist"
Sep 16 13:15:51.524711 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.524687    2579 scope.go:117] "RemoveContainer" containerID="bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"
Sep 16 13:15:51.524909 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:15:51.524888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0\": container with ID starting with bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0 not found: ID does not exist" containerID="bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"
Sep 16 13:15:51.524976 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.524918    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0"} err="failed to get container status \"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0\": rpc error: code = NotFound desc = could not find container \"bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0\": container with ID starting with bb8c8c8521e8e4161b2318814ddb006473ad4c56631fd000caaedbe6b967a8c0 not found: ID does not exist"
Sep 16 13:15:51.531445 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.531421    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:15:51.534512 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:51.534492    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-78bcf54b99-dfmrk"]
Sep 16 13:15:52.458234 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:52.458204    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" path="/var/lib/kubelet/pods/d49d8f8b-5546-4d35-b0ec-e97dab388471/volumes"
Sep 16 13:15:53.896555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:15:53.896515    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:03.896236 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:03.896191    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:03.896713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:03.896303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:16:13.896380 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:13.896345    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:20.200147 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.200120    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:16:20.352307 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.352222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57w4m\" (UniqueName: \"kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m\") pod \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") "
Sep 16 13:16:20.352307 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.352286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs\") pod \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\" (UID: \"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d\") "
Sep 16 13:16:20.354322 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.354289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m" (OuterVolumeSpecName: "kube-api-access-57w4m") pod "b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" (UID: "b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d"). InnerVolumeSpecName "kube-api-access-57w4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:20.354437 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.354379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" (UID: "b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:20.452894 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.452866    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57w4m\" (UniqueName: \"kubernetes.io/projected/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-kube-api-access-57w4m\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:16:20.452894 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.452890    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:16:20.630489 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.630393    2579 generic.go:356] "Generic (PLEG): container finished" podID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerID="20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0" exitCode=0
Sep 16 13:16:20.630489 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.630440    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"
Sep 16 13:16:20.630489 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.630453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" event={"ID":"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d","Type":"ContainerDied","Data":"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"}
Sep 16 13:16:20.630743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.630509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb" event={"ID":"b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d","Type":"ContainerDied","Data":"c866022a523149c3328c23373b50cb1239369dadd282bdcff3ee6b8c7f560780"}
Sep 16 13:16:20.630743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.630540    2579 scope.go:117] "RemoveContainer" containerID="20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"
Sep 16 13:16:20.638441 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.638423    2579 scope.go:117] "RemoveContainer" containerID="20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"
Sep 16 13:16:20.638720 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:16:20.638701    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0\": container with ID starting with 20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0 not found: ID does not exist" containerID="20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"
Sep 16 13:16:20.638792 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.638725    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0"} err="failed to get container status \"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0\": rpc error: code = NotFound desc = could not find container \"20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0\": container with ID starting with 20c11679aa4dfa560830c2accbea390676805233dfe367bd990b1de691044ba0 not found: ID does not exist"
Sep 16 13:16:20.647413 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.647389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:16:20.650876 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:20.650856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-jwxfb"]
Sep 16 13:16:22.458534 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:22.458503    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" path="/var/lib/kubelet/pods/b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d/volumes"
Sep 16 13:16:30.982213 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982177    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982475    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982487    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982497    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982502    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982513    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982518    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982537    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="storage-initializer"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982543    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="storage-initializer"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982614    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44a2a8f-fa12-45c0-9c84-eb7edbcf6a3d" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982622    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49d8f8b-5546-4d35-b0ec-e97dab388471" containerName="main"
Sep 16 13:16:30.982665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.982629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b58c7a5-dc67-4cf5-a845-c0bb27d4b71d" containerName="main"
Sep 16 13:16:30.986788 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.986765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:30.990239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.990208    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 13:16:30.990239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.990224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 13:16:30.990424 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.990269    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-rxzlt\""
Sep 16 13:16:30.990424 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.990214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 13:16:30.997903 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:30.997880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:16:31.141160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.141160 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22kh\" (UniqueName: \"kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.141374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.141374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.141374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.141374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.141350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242695 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22kh\" (UniqueName: \"kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242695 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.242902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.242886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.244778 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.244761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.244871 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.244853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.249468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.249450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22kh\" (UniqueName: \"kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh\") pod \"canary-v1-kserve-65d58ccc9-27g2f\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.296754 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.296720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:31.572427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.572399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:16:31.578418 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.578395    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:16:31.678706 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:31.678665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" event={"ID":"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69","Type":"ContainerStarted","Data":"2a74ca96f994727420b8002ce63f5d774580be5ca3e97b07a43a693cad5e23a6"}
Sep 16 13:16:33.690120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:33.690085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" event={"ID":"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69","Type":"ContainerStarted","Data":"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"}
Sep 16 13:16:33.708264 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:33.708215    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" podStartSLOduration=2.487475736 podStartE2EDuration="3.708198193s" totalImagesPullingTime="1.220722457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:33.70745945 +0000 UTC m=+1699.746258573" watchObservedRunningTime="2026-09-16 13:16:33.708198193 +0000 UTC m=+1699.746997315"
Sep 16 13:16:34.297372 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:34.297338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:41.297863 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:41.297785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:41.310300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:41.310269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:16:41.740322 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:16:41.740295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:17:07.841644 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.841611    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:17:07.843816 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.843799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.847552 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.847525    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 13:17:07.847708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.847573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 13:17:07.847708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.847526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 13:17:07.847708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.847526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-8l8zm\""
Sep 16 13:17:07.854685 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.854661    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:17:07.955374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.955374 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.955609 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.955609 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.955609 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:07.955609 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:07.955541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22982\" (UniqueName: \"kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.040619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.040581    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:17:08.042871 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.042855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.045632 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.045609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-28m9z\""
Sep 16 13:17:08.056441 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.056603 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.056603 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22982\" (UniqueName: \"kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.056603 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.056603 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.056863 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.056657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.057071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.057046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.057205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.057182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.057470 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.057448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.058450 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.058429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:17:08.059669 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.059637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.059935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.059915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.065452 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.065431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22982\" (UniqueName: \"kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.154445 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.154414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:08.157856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.157830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlr5v\" (UniqueName: \"kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.157914 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.157899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.258724 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.258678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr5v\" (UniqueName: \"kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.258913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.258741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.261332 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.261308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.267739 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.267717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr5v\" (UniqueName: \"kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.353748 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.353699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.427487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.427418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:17:08.431399 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:17:08.431375    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c19be6_564f_4fee_8a11_96c94d2a0e13.slice/crio-d05cb6a8050bbe0b6aa9cc626ba919d89a73cf9e59722f9ee4a88c557b3be28b WatchSource:0}: Error finding container d05cb6a8050bbe0b6aa9cc626ba919d89a73cf9e59722f9ee4a88c557b3be28b: Status 404 returned error can't find the container with id d05cb6a8050bbe0b6aa9cc626ba919d89a73cf9e59722f9ee4a88c557b3be28b
Sep 16 13:17:08.623586 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.623527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:17:08.626499 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:17:08.626470    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d52b38_d7be_4850_8534_8a2fc933ec23.slice/crio-ab7606b522e910b1a61e3e3d0208a39f08f43f46db951446ac9d90111e11a165 WatchSource:0}: Error finding container ab7606b522e910b1a61e3e3d0208a39f08f43f46db951446ac9d90111e11a165: Status 404 returned error can't find the container with id ab7606b522e910b1a61e3e3d0208a39f08f43f46db951446ac9d90111e11a165
Sep 16 13:17:08.845083 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.844941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" event={"ID":"42c19be6-564f-4fee-8a11-96c94d2a0e13","Type":"ContainerStarted","Data":"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"}
Sep 16 13:17:08.845083 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.845015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" event={"ID":"42c19be6-564f-4fee-8a11-96c94d2a0e13","Type":"ContainerStarted","Data":"d05cb6a8050bbe0b6aa9cc626ba919d89a73cf9e59722f9ee4a88c557b3be28b"}
Sep 16 13:17:08.846652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.846628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" event={"ID":"30d52b38-d7be-4850-8534-8a2fc933ec23","Type":"ContainerStarted","Data":"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"}
Sep 16 13:17:08.846760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.846670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" event={"ID":"30d52b38-d7be-4850-8534-8a2fc933ec23","Type":"ContainerStarted","Data":"ab7606b522e910b1a61e3e3d0208a39f08f43f46db951446ac9d90111e11a165"}
Sep 16 13:17:08.846802 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.846769    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:17:08.865994 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.865928    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" podStartSLOduration=1.86590656 podStartE2EDuration="1.86590656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:08.864161201 +0000 UTC m=+1734.902960323" watchObservedRunningTime="2026-09-16 13:17:08.86590656 +0000 UTC m=+1734.904705684"
Sep 16 13:17:08.882284 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:08.882222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podStartSLOduration=0.882206338 podStartE2EDuration="882.206338ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:08.881205633 +0000 UTC m=+1734.920004767" watchObservedRunningTime="2026-09-16 13:17:08.882206338 +0000 UTC m=+1734.921005459"
Sep 16 13:17:09.155282 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:09.155251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:18.155104 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:18.155065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:18.168315 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:18.168283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:18.900173 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:18.900145    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:17:39.852281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:17:39.852250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:18:14.457300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:14.457268    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:18:14.460101 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:14.460078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:18:40.267701 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:40.267659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:18:55.286815 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.286752    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" podUID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" containerName="main" containerID="cri-o://e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638" gracePeriod=45
Sep 16 13:18:55.539782 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.539705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:18:55.601841 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.601801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.601876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.601903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t22kh\" (UniqueName: \"kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.601928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.601970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.602005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache\") pod \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\" (UID: \"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69\") "
Sep 16 13:18:55.602300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.602209    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:55.602462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.602428    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:55.602626 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.602435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home" (OuterVolumeSpecName: "home") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:55.604372 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.604342    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh" (OuterVolumeSpecName: "kube-api-access-t22kh") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "kube-api-access-t22kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:18:55.604666 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.604642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm" (OuterVolumeSpecName: "dshm") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:55.604666 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.604649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" (UID: "5b7b7dd3-71ee-49fc-b3a2-1b63901eca69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:18:55.702974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.702932    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22kh\" (UniqueName: \"kubernetes.io/projected/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-kube-api-access-t22kh\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:55.702974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.702967    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:55.702974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.702978    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:55.703236 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.702986    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:55.703236 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.703000    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:55.703236 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:55.703011    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:18:56.306013 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.305976    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" containerID="e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638" exitCode=0
Sep 16 13:18:56.306499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.306020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"
Sep 16 13:18:56.306499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.306044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" event={"ID":"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69","Type":"ContainerDied","Data":"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"}
Sep 16 13:18:56.306499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.306097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f" event={"ID":"5b7b7dd3-71ee-49fc-b3a2-1b63901eca69","Type":"ContainerDied","Data":"2a74ca96f994727420b8002ce63f5d774580be5ca3e97b07a43a693cad5e23a6"}
Sep 16 13:18:56.306499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.306129    2579 scope.go:117] "RemoveContainer" containerID="e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"
Sep 16 13:18:56.314843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.314820    2579 scope.go:117] "RemoveContainer" containerID="e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"
Sep 16 13:18:56.315100 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:18:56.315082    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638\": container with ID starting with e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638 not found: ID does not exist" containerID="e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"
Sep 16 13:18:56.315164 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.315112    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638"} err="failed to get container status \"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638\": rpc error: code = NotFound desc = could not find container \"e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638\": container with ID starting with e2fff3d4a91c604a08bb806ed2a6159a30153d837230d7f3011412f17af89638 not found: ID does not exist"
Sep 16 13:18:56.334622 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.334582    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:18:56.345224 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.341693    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-27g2f"]
Sep 16 13:18:56.457683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:18:56.457654    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" path="/var/lib/kubelet/pods/5b7b7dd3-71ee-49fc-b3a2-1b63901eca69/volumes"
Sep 16 13:19:18.981249 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:18.981213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:19:18.990117 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:19:18.990081    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:19:18.990117 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:18.990117 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:18.990117 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408" cmd=["/bin/sleep","15"]
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:18.990136    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" message="" err=<
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:19:18.990155    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerName="main" containerID="cri-o://3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"
Sep 16 13:19:18.990420 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:18.990199    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerName="main" containerID="cri-o://3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408" gracePeriod=60
Sep 16 13:19:18.997399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:18.997372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:19:19.349026 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.349000    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:19:19.406718 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.406674    2579 generic.go:356] "Generic (PLEG): container finished" podID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerID="3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408" exitCode=0
Sep 16 13:19:19.406915 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.406738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" event={"ID":"42c19be6-564f-4fee-8a11-96c94d2a0e13","Type":"ContainerDied","Data":"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"}
Sep 16 13:19:19.406915 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.406750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"
Sep 16 13:19:19.406915 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.406795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd" event={"ID":"42c19be6-564f-4fee-8a11-96c94d2a0e13","Type":"ContainerDied","Data":"d05cb6a8050bbe0b6aa9cc626ba919d89a73cf9e59722f9ee4a88c557b3be28b"}
Sep 16 13:19:19.406915 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.406823    2579 scope.go:117] "RemoveContainer" containerID="3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"
Sep 16 13:19:19.415316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415365    2579 scope.go:117] "RemoveContainer" containerID="3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"
Sep 16 13:19:19.415468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22982\" (UniqueName: \"kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982\") pod \"42c19be6-564f-4fee-8a11-96c94d2a0e13\" (UID: \"42c19be6-564f-4fee-8a11-96c94d2a0e13\") "
Sep 16 13:19:19.415703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home" (OuterVolumeSpecName: "home") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.415812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache" (OuterVolumeSpecName: "model-cache") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.415812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415791    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.415923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415789    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.415923 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:19:19.415874    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408\": container with ID starting with 3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408 not found: ID does not exist" containerID="3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"
Sep 16 13:19:19.415923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.415907    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408"} err="failed to get container status \"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408\": rpc error: code = NotFound desc = could not find container \"3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408\": container with ID starting with 3e645e565755cee7a49b08dffcba73847e66661e24f3e51631158842801c7408 not found: ID does not exist"
Sep 16 13:19:19.418035 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.417997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm" (OuterVolumeSpecName: "dshm") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.418136 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.418084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982" (OuterVolumeSpecName: "kube-api-access-22982") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "kube-api-access-22982". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:19:19.418136 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.418084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42c19be6-564f-4fee-8a11-96c94d2a0e13" (UID: "42c19be6-564f-4fee-8a11-96c94d2a0e13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:19:19.516865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.516482    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42c19be6-564f-4fee-8a11-96c94d2a0e13-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.516865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.516525    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.516865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.516538    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.516865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.516551    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42c19be6-564f-4fee-8a11-96c94d2a0e13-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.516865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.516590    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22982\" (UniqueName: \"kubernetes.io/projected/42c19be6-564f-4fee-8a11-96c94d2a0e13-kube-api-access-22982\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.737500 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.737467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:19:19.742137 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.742108    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-rwbzd"]
Sep 16 13:19:19.852045 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:19.851951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:19:20.457479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:20.457434    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" path="/var/lib/kubelet/pods/42c19be6-564f-4fee-8a11-96c94d2a0e13/volumes"
Sep 16 13:19:29.853121 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:29.853078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:19:34.015753 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:34.015706    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" containerID="cri-o://43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758" gracePeriod=45
Sep 16 13:19:39.852501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:39.852419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:19:39.852891 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:39.852534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:19:49.852197 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:49.852160    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:19:59.852858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:19:59.852812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:20:04.274419 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.274394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:20:04.315484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.315395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlr5v\" (UniqueName: \"kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v\") pod \"30d52b38-d7be-4850-8534-8a2fc933ec23\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") "
Sep 16 13:20:04.315484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.315460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs\") pod \"30d52b38-d7be-4850-8534-8a2fc933ec23\" (UID: \"30d52b38-d7be-4850-8534-8a2fc933ec23\") "
Sep 16 13:20:04.317598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.317543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30d52b38-d7be-4850-8534-8a2fc933ec23" (UID: "30d52b38-d7be-4850-8534-8a2fc933ec23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:20:04.317708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.317633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v" (OuterVolumeSpecName: "kube-api-access-qlr5v") pod "30d52b38-d7be-4850-8534-8a2fc933ec23" (UID: "30d52b38-d7be-4850-8534-8a2fc933ec23"). InnerVolumeSpecName "kube-api-access-qlr5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:20:04.416313 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.416274    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30d52b38-d7be-4850-8534-8a2fc933ec23-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:20:04.416313 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.416307    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlr5v\" (UniqueName: \"kubernetes.io/projected/30d52b38-d7be-4850-8534-8a2fc933ec23-kube-api-access-qlr5v\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:20:04.593683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.593588    2579 generic.go:356] "Generic (PLEG): container finished" podID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerID="43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758" exitCode=0
Sep 16 13:20:04.593683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.593619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"
Sep 16 13:20:04.593683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.593660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" event={"ID":"30d52b38-d7be-4850-8534-8a2fc933ec23","Type":"ContainerDied","Data":"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"}
Sep 16 13:20:04.593900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.593713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g" event={"ID":"30d52b38-d7be-4850-8534-8a2fc933ec23","Type":"ContainerDied","Data":"ab7606b522e910b1a61e3e3d0208a39f08f43f46db951446ac9d90111e11a165"}
Sep 16 13:20:04.593900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.593734    2579 scope.go:117] "RemoveContainer" containerID="43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"
Sep 16 13:20:04.601905 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.601884    2579 scope.go:117] "RemoveContainer" containerID="43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"
Sep 16 13:20:04.602180 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:20:04.602157    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758\": container with ID starting with 43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758 not found: ID does not exist" containerID="43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"
Sep 16 13:20:04.602239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.602187    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758"} err="failed to get container status \"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758\": rpc error: code = NotFound desc = could not find container \"43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758\": container with ID starting with 43b1eeb472490f59907a3fec3a2663fab437f75d3c6dac13f5198a14b2c07758 not found: ID does not exist"
Sep 16 13:20:04.624862 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.624839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:20:04.629209 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:04.629189    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75wf69g"]
Sep 16 13:20:06.459502 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:06.459212    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" path="/var/lib/kubelet/pods/30d52b38-d7be-4850-8534-8a2fc933ec23/volumes"
Sep 16 13:20:29.039337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039297    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039587    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039613    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039619    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039681    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d52b38-d7be-4850-8534-8a2fc933ec23" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c19be6-564f-4fee-8a11-96c94d2a0e13" containerName="main"
Sep 16 13:20:29.039734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.039697    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b7b7dd3-71ee-49fc-b3a2-1b63901eca69" containerName="main"
Sep 16 13:20:29.042655 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.042638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.055728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.055694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 13:20:29.055728 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.055694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-w8dqc\""
Sep 16 13:20:29.055907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.055765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-ls7zr\""
Sep 16 13:20:29.055907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.055789    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 13:20:29.055907 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.055729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 13:20:29.061741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.061689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:20:29.130539 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtzg\" (UniqueName: \"kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.130741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.130741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.130741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.130741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.130741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.130717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.231859 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.231823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.231869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.231897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.231917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.231990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtzg\" (UniqueName: \"kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.232058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232343 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.232287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232412 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.232398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.232578 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.232544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.234373 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.234341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.234713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.234694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.242644 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.242625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtzg\" (UniqueName: \"kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.351620 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.351521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:29.385720 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.385685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:20:29.389309 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.389274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.392315 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.392236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-4l27h\""
Sep 16 13:20:29.404110 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.403835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:20:29.534776 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.534739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.534980 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.534869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdr4\" (UniqueName: \"kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.633774 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.633587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:20:29.636146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.636123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdr4\" (UniqueName: \"kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.636270 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.636189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.636431 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:20:29.636404    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19e9029_4c7f_48eb_b566_dfba7488e313.slice/crio-6a8403b94b0148cd9df114ee28b506a5cd62d33d79960a3a1b34968043cd6aa3 WatchSource:0}: Error finding container 6a8403b94b0148cd9df114ee28b506a5cd62d33d79960a3a1b34968043cd6aa3: Status 404 returned error can't find the container with id 6a8403b94b0148cd9df114ee28b506a5cd62d33d79960a3a1b34968043cd6aa3
Sep 16 13:20:29.638912 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.638894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.646196 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.646173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdr4\" (UniqueName: \"kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk59rp\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.699364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.699326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerStarted","Data":"6a8403b94b0148cd9df114ee28b506a5cd62d33d79960a3a1b34968043cd6aa3"}
Sep 16 13:20:29.702588 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.702547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:29.979613 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:29.979575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:20:29.983415 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:20:29.983379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6bf634_22f1_4111_b3df_02519e44448a.slice/crio-076a0bbc2713e6dbfbd538b6af56a32d167336c86bafcc6b4740fbdd7fa3213e WatchSource:0}: Error finding container 076a0bbc2713e6dbfbd538b6af56a32d167336c86bafcc6b4740fbdd7fa3213e: Status 404 returned error can't find the container with id 076a0bbc2713e6dbfbd538b6af56a32d167336c86bafcc6b4740fbdd7fa3213e
Sep 16 13:20:30.705269 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:30.705225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" event={"ID":"7b6bf634-22f1-4111-b3df-02519e44448a","Type":"ContainerStarted","Data":"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"}
Sep 16 13:20:30.705752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:30.705296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" event={"ID":"7b6bf634-22f1-4111-b3df-02519e44448a","Type":"ContainerStarted","Data":"076a0bbc2713e6dbfbd538b6af56a32d167336c86bafcc6b4740fbdd7fa3213e"}
Sep 16 13:20:30.705752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:30.705380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:20:30.729796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:30.729741    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podStartSLOduration=1.729725067 podStartE2EDuration="1.729725067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:20:30.728606548 +0000 UTC m=+1936.767405672" watchObservedRunningTime="2026-09-16 13:20:30.729725067 +0000 UTC m=+1936.768524192"
Sep 16 13:20:31.710979 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:31.710943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerStarted","Data":"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"}
Sep 16 13:20:32.717103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:32.717062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerStarted","Data":"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"}
Sep 16 13:20:32.717605 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:32.717158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:32.748990 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:32.748937    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" podStartSLOduration=2.246127372 podStartE2EDuration="3.748916498s" totalImagesPullingTime="1.502789126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:20:32.746404014 +0000 UTC m=+1938.785203137" watchObservedRunningTime="2026-09-16 13:20:32.748916498 +0000 UTC m=+1938.787715621"
Sep 16 13:20:33.352507 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:33.352470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:39.352640 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:39.352596    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:39.365342 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:39.365303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:39.758068 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:39.758041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:20:50.763848 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:20:50.763819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:21:01.713050 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:01.713020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:21:19.454583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.454527    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:21:19.463191 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:21:19.463158    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:21:19.463191 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.463191 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.463191 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537" cmd=["/bin/sleep","15"]
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.463204    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" message="" err=<
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:21:19.463225    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main" containerID="cri-o://1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"
Sep 16 13:21:19.463443 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.463274    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main" containerID="cri-o://1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537" gracePeriod=60
Sep 16 13:21:19.468949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.468921    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:21:19.747715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.747599    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8001/health\": dial tcp 10.134.0.45:8001: connect: connection refused"
Sep 16 13:21:19.911092 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.911056    2579 generic.go:356] "Generic (PLEG): container finished" podID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerID="1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537" exitCode=0
Sep 16 13:21:19.911272 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:19.911134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerDied","Data":"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"}
Sep 16 13:21:20.076740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.076707    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="llm-d-routing-sidecar" containerID="cri-o://a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453" gracePeriod=60
Sep 16 13:21:20.320149 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.320126    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:21:20.436351 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436351 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhtzg\" (UniqueName: \"kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache\") pod \"a19e9029-4c7f-48eb-b566-dfba7488e313\" (UID: \"a19e9029-4c7f-48eb-b566-dfba7488e313\") "
Sep 16 13:21:20.436867 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home" (OuterVolumeSpecName: "home") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.436867 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.436976 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.436952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache" (OuterVolumeSpecName: "model-cache") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.438749 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.438727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm" (OuterVolumeSpecName: "dshm") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.438749 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.438741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg" (OuterVolumeSpecName: "kube-api-access-nhtzg") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "kube-api-access-nhtzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:21:20.438847 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.438733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a19e9029-4c7f-48eb-b566-dfba7488e313" (UID: "a19e9029-4c7f-48eb-b566-dfba7488e313"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:21:20.538006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.537968    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.538006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.538001    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtzg\" (UniqueName: \"kubernetes.io/projected/a19e9029-4c7f-48eb-b566-dfba7488e313-kube-api-access-nhtzg\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.538006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.538010    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e9029-4c7f-48eb-b566-dfba7488e313-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.538462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.538018    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.538462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.538027    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.538462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.538034    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e9029-4c7f-48eb-b566-dfba7488e313-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.916592 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.916540    2579 generic.go:356] "Generic (PLEG): container finished" podID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerID="a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453" exitCode=0
Sep 16 13:21:20.916777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.916623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerDied","Data":"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"}
Sep 16 13:21:20.916777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.916636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"
Sep 16 13:21:20.916777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.916679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2" event={"ID":"a19e9029-4c7f-48eb-b566-dfba7488e313","Type":"ContainerDied","Data":"6a8403b94b0148cd9df114ee28b506a5cd62d33d79960a3a1b34968043cd6aa3"}
Sep 16 13:21:20.916777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.916701    2579 scope.go:117] "RemoveContainer" containerID="1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"
Sep 16 13:21:20.925509 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.925492    2579 scope.go:117] "RemoveContainer" containerID="a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"
Sep 16 13:21:20.932887 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.932865    2579 scope.go:117] "RemoveContainer" containerID="1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"
Sep 16 13:21:20.933148 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:21:20.933120    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537\": container with ID starting with 1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537 not found: ID does not exist" containerID="1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"
Sep 16 13:21:20.933204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.933157    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537"} err="failed to get container status \"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537\": rpc error: code = NotFound desc = could not find container \"1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537\": container with ID starting with 1585d84d1dfa2130b287f395008a402101a3beba4c28b5253d4d2f6ed3b04537 not found: ID does not exist"
Sep 16 13:21:20.933204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.933175    2579 scope.go:117] "RemoveContainer" containerID="a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"
Sep 16 13:21:20.933442 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:21:20.933422    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453\": container with ID starting with a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453 not found: ID does not exist" containerID="a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"
Sep 16 13:21:20.933498 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.933451    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453"} err="failed to get container status \"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453\": rpc error: code = NotFound desc = could not find container \"a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453\": container with ID starting with a29be0651f1db7408ce52b3b7c21fbbbabddcd2bfc375c644f376691c9a25453 not found: ID does not exist"
Sep 16 13:21:20.936602 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.936581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:21:20.939156 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:20.939137    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-pskc2"]
Sep 16 13:21:21.713245 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:21.713208    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:21:22.457972 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:22.457937    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" path="/var/lib/kubelet/pods/a19e9029-4c7f-48eb-b566-dfba7488e313/volumes"
Sep 16 13:21:31.713285 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:31.713245    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:21:34.487107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:34.487068    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" containerID="cri-o://1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975" gracePeriod=45
Sep 16 13:21:41.712951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:41.712908    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:21:41.713396 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:41.713056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:21:51.713032 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:21:51.712993    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:22:01.713418 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:01.713381    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:22:04.736543 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.736511    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:22:04.790619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.790514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gdr4\" (UniqueName: \"kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4\") pod \"7b6bf634-22f1-4111-b3df-02519e44448a\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") "
Sep 16 13:22:04.790619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.790554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs\") pod \"7b6bf634-22f1-4111-b3df-02519e44448a\" (UID: \"7b6bf634-22f1-4111-b3df-02519e44448a\") "
Sep 16 13:22:04.792759 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.792718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b6bf634-22f1-4111-b3df-02519e44448a" (UID: "7b6bf634-22f1-4111-b3df-02519e44448a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:22:04.792759 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.792727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4" (OuterVolumeSpecName: "kube-api-access-8gdr4") pod "7b6bf634-22f1-4111-b3df-02519e44448a" (UID: "7b6bf634-22f1-4111-b3df-02519e44448a"). InnerVolumeSpecName "kube-api-access-8gdr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:04.891883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.891849    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gdr4\" (UniqueName: \"kubernetes.io/projected/7b6bf634-22f1-4111-b3df-02519e44448a-kube-api-access-8gdr4\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:22:04.891883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:04.891881    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6bf634-22f1-4111-b3df-02519e44448a-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:22:05.100129 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.100036    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b6bf634-22f1-4111-b3df-02519e44448a" containerID="1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975" exitCode=0
Sep 16 13:22:05.100129 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.100100    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"
Sep 16 13:22:05.100318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.100124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" event={"ID":"7b6bf634-22f1-4111-b3df-02519e44448a","Type":"ContainerDied","Data":"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"}
Sep 16 13:22:05.100318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.100207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp" event={"ID":"7b6bf634-22f1-4111-b3df-02519e44448a","Type":"ContainerDied","Data":"076a0bbc2713e6dbfbd538b6af56a32d167336c86bafcc6b4740fbdd7fa3213e"}
Sep 16 13:22:05.100318 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.100229    2579 scope.go:117] "RemoveContainer" containerID="1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"
Sep 16 13:22:05.109091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.109068    2579 scope.go:117] "RemoveContainer" containerID="1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"
Sep 16 13:22:05.109388 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:22:05.109370    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975\": container with ID starting with 1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975 not found: ID does not exist" containerID="1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"
Sep 16 13:22:05.109434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.109397    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975"} err="failed to get container status \"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975\": rpc error: code = NotFound desc = could not find container \"1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975\": container with ID starting with 1f9195dcd2606a97df69e9aaaf07cc0108d8d6df197dce8ffc1df6e87fa7c975 not found: ID does not exist"
Sep 16 13:22:05.122488 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.122461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:22:05.126771 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:05.126748    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk59rp"]
Sep 16 13:22:06.457844 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:06.457814    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" path="/var/lib/kubelet/pods/7b6bf634-22f1-4111-b3df-02519e44448a/volumes"
Sep 16 13:22:21.899777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.899735    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:22:21.900181 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900156    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="llm-d-routing-sidecar"
Sep 16 13:22:21.900181 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900176    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="llm-d-routing-sidecar"
Sep 16 13:22:21.900287 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900200    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main"
Sep 16 13:22:21.900287 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900210    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main"
Sep 16 13:22:21.900287 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900225    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main"
Sep 16 13:22:21.900287 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900234    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main"
Sep 16 13:22:21.900464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900324    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="llm-d-routing-sidecar"
Sep 16 13:22:21.900464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900335    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6bf634-22f1-4111-b3df-02519e44448a" containerName="main"
Sep 16 13:22:21.900464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.900345    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19e9029-4c7f-48eb-b566-dfba7488e313" containerName="main"
Sep 16 13:22:21.903092 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.903076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:21.905742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.905718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 13:22:21.905883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.905763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:22:21.906824 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.906803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 13:22:21.907232 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.907214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 13:22:21.907331 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.907222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-5rj8s\""
Sep 16 13:22:21.907331 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.907298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 13:22:21.918189 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:21.918164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:22:22.011483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jqx\" (UniqueName: \"kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.011752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.011716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.112932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.112891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.112932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.112936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jqx\" (UniqueName: \"kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.112970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113587 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113672 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113719 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113719 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.113971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.113949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.115413 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.115394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.115710 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.115693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.120665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.120640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jqx\" (UniqueName: \"kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.173075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.172987    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:22:22.177295 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.177271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.189985 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.189957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:22:22.213350 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.213315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:22.213791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.213764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.214098 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.214021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.214211 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.214111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.214211 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.214167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315467 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315747 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.315976 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.315961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.316306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.316285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.318097 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.318074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.487103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.487010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:22:22.487103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.487076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:22:22.490399 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:22:22.490370    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c0581d_461e_40a8_b66a_562f35bacf3f.slice/crio-c0390a1c9374e29116fd35013b8830f7e0dbec1e4795ad3a6f0613df533c2722 WatchSource:0}: Error finding container c0390a1c9374e29116fd35013b8830f7e0dbec1e4795ad3a6f0613df533c2722: Status 404 returned error can't find the container with id c0390a1c9374e29116fd35013b8830f7e0dbec1e4795ad3a6f0613df533c2722
Sep 16 13:22:22.761721 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:22.761693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:22:22.763272 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:22:22.763243    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a25d24_e81f_4c2b_983b_afb2a331fa3d.slice/crio-a4441c5ddab7ef0562b486eb4788dbed0db5ff0ddca5e3851310bee0c09fe261 WatchSource:0}: Error finding container a4441c5ddab7ef0562b486eb4788dbed0db5ff0ddca5e3851310bee0c09fe261: Status 404 returned error can't find the container with id a4441c5ddab7ef0562b486eb4788dbed0db5ff0ddca5e3851310bee0c09fe261
Sep 16 13:22:23.180421 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:23.180373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerStarted","Data":"20ae3c9a8e62e660fc8359f9ce43ee6a07c954e9a4be85ea28bccdd8db34834e"}
Sep 16 13:22:23.180942 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:23.180489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerStarted","Data":"c0390a1c9374e29116fd35013b8830f7e0dbec1e4795ad3a6f0613df533c2722"}
Sep 16 13:22:23.182662 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:23.182625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerStarted","Data":"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"}
Sep 16 13:22:23.182805 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:23.182680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerStarted","Data":"a4441c5ddab7ef0562b486eb4788dbed0db5ff0ddca5e3851310bee0c09fe261"}
Sep 16 13:22:25.194601 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:25.194548    2579 generic.go:356] "Generic (PLEG): container finished" podID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerID="20ae3c9a8e62e660fc8359f9ce43ee6a07c954e9a4be85ea28bccdd8db34834e" exitCode=0
Sep 16 13:22:25.195063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:25.194598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerDied","Data":"20ae3c9a8e62e660fc8359f9ce43ee6a07c954e9a4be85ea28bccdd8db34834e"}
Sep 16 13:22:26.201129 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.201089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerStarted","Data":"2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d"}
Sep 16 13:22:26.202646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.202621    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerID="9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba" exitCode=0
Sep 16 13:22:26.202764 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.202692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerDied","Data":"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"}
Sep 16 13:22:26.203743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.203727    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:22:26.213530 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.213514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:26.222018 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:26.221978    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" podStartSLOduration=2.623930191 podStartE2EDuration="5.2219661s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.598035909s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:26.220504102 +0000 UTC m=+2052.259303226" watchObservedRunningTime="2026-09-16 13:22:26.2219661 +0000 UTC m=+2052.260765223"
Sep 16 13:22:32.213863 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:32.213534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:32.231343 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:32.231274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:22:32.259128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:22:32.259095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:23:13.455665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:13.455630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerStarted","Data":"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"}
Sep 16 13:23:13.456124 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:13.455839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:23:13.474214 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:13.474166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" podStartSLOduration=2.67146289 podStartE2EDuration="51.474153077s" totalImagesPullingTime="46.238023358s" totalInitContainerRuntime="2.564666829s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:13.473598744 +0000 UTC m=+2099.512397866" watchObservedRunningTime="2026-09-16 13:23:13.474153077 +0000 UTC m=+2099.512952244"
Sep 16 13:23:14.490969 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:14.490929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:23:14.494139 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:14.494113    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:23:44.473460 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:44.473428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:23:46.000168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.000136    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b_c8a25d24-e81f-4c2b-983b-afb2a331fa3d/main/0.log"
Sep 16 13:23:46.176358 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.176322    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:23:46.176673 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.176618    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="main" containerID="cri-o://4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5" gracePeriod=30
Sep 16 13:23:46.192000 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.191969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:23:46.200508 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:23:46.200476    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:23:46.200508 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.200508 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.200508 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d" cmd=["/bin/sleep","15"]
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.200526    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" message="" err=<
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:23:46.200538    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main" containerID="cri-o://2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d"
Sep 16 13:23:46.200800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.200603    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main" containerID="cri-o://2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d" gracePeriod=60
Sep 16 13:23:46.242467 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.242431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 16 13:23:46.600660 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.600622    2579 generic.go:356] "Generic (PLEG): container finished" podID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerID="2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d" exitCode=0
Sep 16 13:23:46.600778 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.600691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerDied","Data":"2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d"}
Sep 16 13:23:46.851066 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.851030    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:23:46.986221 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir\") pod \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") "
Sep 16 13:23:46.986221 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location\") pod \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") "
Sep 16 13:23:46.986470 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs\") pod \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") "
Sep 16 13:23:46.986470 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert\") pod \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\" (UID: \"c8a25d24-e81f-4c2b-983b-afb2a331fa3d\") "
Sep 16 13:23:46.986688 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8a25d24-e81f-4c2b-983b-afb2a331fa3d" (UID: "c8a25d24-e81f-4c2b-983b-afb2a331fa3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:46.986763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.986748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8a25d24-e81f-4c2b-983b-afb2a331fa3d" (UID: "c8a25d24-e81f-4c2b-983b-afb2a331fa3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:46.988530 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.988506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a25d24-e81f-4c2b-983b-afb2a331fa3d" (UID: "c8a25d24-e81f-4c2b-983b-afb2a331fa3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:46.994552 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:46.994515    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:23:47.007472 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.007439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8a25d24-e81f-4c2b-983b-afb2a331fa3d" (UID: "c8a25d24-e81f-4c2b-983b-afb2a331fa3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.087367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087367 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96jqx\" (UniqueName: \"kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location\") pod \"40c0581d-461e-40a8-b66a-562f35bacf3f\" (UID: \"40c0581d-461e-40a8-b66a-562f35bacf3f\") "
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087658    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:47.087740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087736    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home" (OuterVolumeSpecName: "home") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087910    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c0581d-461e-40a8-b66a-562f35bacf3f-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087930    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087949    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087965    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087986    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.087998    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.088010    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.088111 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.088022    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a25d24-e81f-4c2b-983b-afb2a331fa3d-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.089604 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.089578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm" (OuterVolumeSpecName: "dshm") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.089725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.089647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx" (OuterVolumeSpecName: "kube-api-access-96jqx") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "kube-api-access-96jqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:23:47.089772 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.089748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:47.108487 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.108445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40c0581d-461e-40a8-b66a-562f35bacf3f" (UID: "40c0581d-461e-40a8-b66a-562f35bacf3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.189323 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.189283    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jqx\" (UniqueName: \"kubernetes.io/projected/40c0581d-461e-40a8-b66a-562f35bacf3f-kube-api-access-96jqx\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.189323 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.189316    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c0581d-461e-40a8-b66a-562f35bacf3f-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.189323 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.189328    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.189581 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.189336    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c0581d-461e-40a8-b66a-562f35bacf3f-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.605760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.605731    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerID="4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5" exitCode=0
Sep 16 13:23:47.605946 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.605851    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"
Sep 16 13:23:47.605946 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.605868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerDied","Data":"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"}
Sep 16 13:23:47.606046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.605943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b" event={"ID":"c8a25d24-e81f-4c2b-983b-afb2a331fa3d","Type":"ContainerDied","Data":"a4441c5ddab7ef0562b486eb4788dbed0db5ff0ddca5e3851310bee0c09fe261"}
Sep 16 13:23:47.606046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.605971    2579 scope.go:117] "RemoveContainer" containerID="4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"
Sep 16 13:23:47.607614 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.607591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl" event={"ID":"40c0581d-461e-40a8-b66a-562f35bacf3f","Type":"ContainerDied","Data":"c0390a1c9374e29116fd35013b8830f7e0dbec1e4795ad3a6f0613df533c2722"}
Sep 16 13:23:47.607705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.607625    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"
Sep 16 13:23:47.615966 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.615938    2579 scope.go:117] "RemoveContainer" containerID="9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"
Sep 16 13:23:47.623607 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.623586    2579 scope.go:117] "RemoveContainer" containerID="4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"
Sep 16 13:23:47.623868 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:23:47.623848    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5\": container with ID starting with 4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5 not found: ID does not exist" containerID="4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"
Sep 16 13:23:47.623951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.623881    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5"} err="failed to get container status \"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5\": rpc error: code = NotFound desc = could not find container \"4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5\": container with ID starting with 4099fd4e012e8d629b80384332dd650a8abb918e99cf60f569589d8c3af064a5 not found: ID does not exist"
Sep 16 13:23:47.623951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.623922    2579 scope.go:117] "RemoveContainer" containerID="9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"
Sep 16 13:23:47.624187 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:23:47.624170    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba\": container with ID starting with 9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba not found: ID does not exist" containerID="9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"
Sep 16 13:23:47.624224 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.624192    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba"} err="failed to get container status \"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba\": rpc error: code = NotFound desc = could not find container \"9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba\": container with ID starting with 9f9de7527a44e0b0b6172b586042e5ac42e293a621f03c31b3234a19114634ba not found: ID does not exist"
Sep 16 13:23:47.624224 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.624205    2579 scope.go:117] "RemoveContainer" containerID="2391974fc903fb25fca41ec156a1e8ba4b391b2237e91d1813df6c1dbf20108d"
Sep 16 13:23:47.631685 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.631638    2579 scope.go:117] "RemoveContainer" containerID="20ae3c9a8e62e660fc8359f9ce43ee6a07c954e9a4be85ea28bccdd8db34834e"
Sep 16 13:23:47.633103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.633086    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:23:47.638229 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.638207    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-vg8gl"]
Sep 16 13:23:47.652234 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.652207    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:23:47.656467 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:47.656442    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86bcfc8b6f-g2g5b"]
Sep 16 13:23:48.458725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:48.458692    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" path="/var/lib/kubelet/pods/40c0581d-461e-40a8-b66a-562f35bacf3f/volumes"
Sep 16 13:23:48.459121 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:23:48.459113    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" path="/var/lib/kubelet/pods/c8a25d24-e81f-4c2b-983b-afb2a331fa3d/volumes"
Sep 16 13:24:13.311368 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311294    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311628    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="storage-initializer"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311642    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="storage-initializer"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311654    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="main"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311661    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="main"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311675    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="storage-initializer"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311682    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="storage-initializer"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311692    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main"
Sep 16 13:24:13.311725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311698    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main"
Sep 16 13:24:13.311967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311761    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c0581d-461e-40a8-b66a-562f35bacf3f" containerName="main"
Sep 16 13:24:13.311967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.311771    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a25d24-e81f-4c2b-983b-afb2a331fa3d" containerName="main"
Sep 16 13:24:13.313471 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.313455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.318003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.317982    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 13:24:13.318151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.317981    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 13:24:13.319097 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.319078    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-c6zv6\""
Sep 16 13:24:13.319170 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.319125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 13:24:13.333155 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.333129    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:24:13.389685 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbns\" (UniqueName: \"kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.389851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.389851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.389851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.389851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.389996 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.389881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491426 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491426 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491426 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbns\" (UniqueName: \"kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491600 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491661 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491661 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.491773 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.491739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.493691 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.493661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.493834 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.493818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.499992 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.499961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbns\" (UniqueName: \"kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns\") pod \"leave-v1-kserve-65684d4f5-wclrx\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.623409 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.623310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:13.905183 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:13.905140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:24:14.723821 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:14.723777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" event={"ID":"6052a105-eb0c-4ed4-bf0d-5159bd14dd01","Type":"ContainerStarted","Data":"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"}
Sep 16 13:24:14.724189 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:14.723840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" event={"ID":"6052a105-eb0c-4ed4-bf0d-5159bd14dd01","Type":"ContainerStarted","Data":"d1643826ed938270996211a787232b02aac977c29a7ce7a07e7f03540a0f4323"}
Sep 16 13:24:14.745043 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:14.744983    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" podStartSLOduration=1.744965336 podStartE2EDuration="1.744965336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:14.743477354 +0000 UTC m=+2160.782276482" watchObservedRunningTime="2026-09-16 13:24:14.744965336 +0000 UTC m=+2160.783764460"
Sep 16 13:24:15.623640 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:15.623601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:21.893116 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.893081    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:24:21.895263 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.895245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.898361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.898341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 13:24:21.954501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.954468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:24:21.959964 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.959934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.960120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.959974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.960120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.960041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd4vm\" (UniqueName: \"kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.960120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.960088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.960120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.960105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:21.960277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:21.960121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061461 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd4vm\" (UniqueName: \"kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.061713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.061649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.062120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.062095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.062191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.062106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.062191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.062134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.063759 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.063739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.064032 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.064015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.071420 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.071396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd4vm\" (UniqueName: \"kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm\") pod \"leave-v2-kserve-556b4dddf9-htps9\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.205464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.205424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:22.480468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.480388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:24:22.484198 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:24:22.484163    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901ecc72_4eb3_49b8_be96_793614191b04.slice/crio-818b79f4e023fce6031425f67b96af9494b6e11505878c3dd475cdb9b392830c WatchSource:0}: Error finding container 818b79f4e023fce6031425f67b96af9494b6e11505878c3dd475cdb9b392830c: Status 404 returned error can't find the container with id 818b79f4e023fce6031425f67b96af9494b6e11505878c3dd475cdb9b392830c
Sep 16 13:24:22.758456 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.758357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" event={"ID":"901ecc72-4eb3-49b8-be96-793614191b04","Type":"ContainerStarted","Data":"b695da51f76fc57fe7694106cf5be16193f3c121069860944ee8fe70e3e5647f"}
Sep 16 13:24:22.758456 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:22.758418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" event={"ID":"901ecc72-4eb3-49b8-be96-793614191b04","Type":"ContainerStarted","Data":"818b79f4e023fce6031425f67b96af9494b6e11505878c3dd475cdb9b392830c"}
Sep 16 13:24:23.206150 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:23.206115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:23.624213 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:23.624128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:23.636555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:23.636528    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:23.658658 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:23.658605    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" podStartSLOduration=2.658583771 podStartE2EDuration="2.658583771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:22.785372965 +0000 UTC m=+2168.824172088" watchObservedRunningTime="2026-09-16 13:24:23.658583771 +0000 UTC m=+2169.697382885"
Sep 16 13:24:23.775319 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:23.775291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:24:32.205577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:32.205521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:32.218459 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:32.218432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:32.814402 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:32.814368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:24:49.050598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.050552    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:24:49.053166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.053146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.057205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.057179    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:24:49.058055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.058038    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 13:24:49.058988 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.058960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-jbv4x\""
Sep 16 13:24:49.059091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.059045    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 13:24:49.059443 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.059425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 13:24:49.059547 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.059448    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 13:24:49.068520 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.068497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:24:49.183277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg2p\" (UniqueName: \"kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.183496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.183461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg2p\" (UniqueName: \"kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284542 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.284913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.284890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.285179 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.285156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.285285 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.285191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.285285 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.285230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.285285 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.285269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.286695 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.286670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.287077 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.287059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.292355 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.292334    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:24:49.295811 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.295787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.299151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.299127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg2p\" (UniqueName: \"kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-7lths\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.319358 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.319329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:24:49.363981 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.363938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:49.385785 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.385756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.385949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.385807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.385949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.385907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.386070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.385966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.486629 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.486587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.486629 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.486629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.486888 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.486663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.486888 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.486704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.487026 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.487004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.487090 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.487017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.487380 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.487358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.489172 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.489154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.616554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.616471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:49.642781 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.642754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:24:49.645199 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:24:49.645165    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb240ec_14fb_44bb_90ee_b6b696b2eefd.slice/crio-521765b023fcf36f98e6994b26c5906a6dd8fb40901f68c8540df7fa1237165a WatchSource:0}: Error finding container 521765b023fcf36f98e6994b26c5906a6dd8fb40901f68c8540df7fa1237165a: Status 404 returned error can't find the container with id 521765b023fcf36f98e6994b26c5906a6dd8fb40901f68c8540df7fa1237165a
Sep 16 13:24:49.881046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.880967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerStarted","Data":"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"}
Sep 16 13:24:49.881046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.881015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerStarted","Data":"521765b023fcf36f98e6994b26c5906a6dd8fb40901f68c8540df7fa1237165a"}
Sep 16 13:24:49.892781 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:49.892740    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:24:49.895165 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:24:49.895138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b097f5_efb4_44b1_8289_dc9ef93da2a5.slice/crio-03740092232d085a545e1e8f02b7893a902416b16dd7319a59a47c78105ccce6 WatchSource:0}: Error finding container 03740092232d085a545e1e8f02b7893a902416b16dd7319a59a47c78105ccce6: Status 404 returned error can't find the container with id 03740092232d085a545e1e8f02b7893a902416b16dd7319a59a47c78105ccce6
Sep 16 13:24:50.887497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:50.887458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerStarted","Data":"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"}
Sep 16 13:24:50.887911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:50.887546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerStarted","Data":"03740092232d085a545e1e8f02b7893a902416b16dd7319a59a47c78105ccce6"}
Sep 16 13:24:52.899232 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:52.899195    2579 generic.go:356] "Generic (PLEG): container finished" podID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerID="e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927" exitCode=0
Sep 16 13:24:52.899671 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:52.899269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerDied","Data":"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"}
Sep 16 13:24:52.900917 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:52.900894    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerID="f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece" exitCode=0
Sep 16 13:24:52.901013 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:52.900955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerDied","Data":"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"}
Sep 16 13:24:53.908209 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:53.908175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerStarted","Data":"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"}
Sep 16 13:24:53.910166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:53.910143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerStarted","Data":"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"}
Sep 16 13:24:53.910361 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:53.910343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:24:53.935213 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:53.935154    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" podStartSLOduration=2.358060609 podStartE2EDuration="4.935136342s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.577075733s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:53.93469688 +0000 UTC m=+2199.973496003" watchObservedRunningTime="2026-09-16 13:24:53.935136342 +0000 UTC m=+2199.973935465"
Sep 16 13:24:53.959376 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:53.959323    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" podStartSLOduration=2.4322249830000002 podStartE2EDuration="4.959308439s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.527083456s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:53.958118328 +0000 UTC m=+2199.996917451" watchObservedRunningTime="2026-09-16 13:24:53.959308439 +0000 UTC m=+2199.998107562"
Sep 16 13:24:54.364310 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:54.364223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:59.364219 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:59.364178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:59.376835 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:59.376810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:24:59.955668 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:24:59.955640    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:25:06.899273 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:06.899235    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:25:07.469622 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:07.469588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:25:21.918300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:21.918232    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" podUID="901ecc72-4eb3-49b8-be96-793614191b04" containerName="main" containerID="cri-o://b695da51f76fc57fe7694106cf5be16193f3c121069860944ee8fe70e3e5647f" gracePeriod=45
Sep 16 13:25:22.045286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.045250    2579 generic.go:356] "Generic (PLEG): container finished" podID="901ecc72-4eb3-49b8-be96-793614191b04" containerID="b695da51f76fc57fe7694106cf5be16193f3c121069860944ee8fe70e3e5647f" exitCode=0
Sep 16 13:25:22.045442 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.045329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" event={"ID":"901ecc72-4eb3-49b8-be96-793614191b04","Type":"ContainerDied","Data":"b695da51f76fc57fe7694106cf5be16193f3c121069860944ee8fe70e3e5647f"}
Sep 16 13:25:22.182179 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.182120    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:25:22.269752 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd4vm\" (UniqueName: \"kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.269925 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.269925 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.269925 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.269925 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.270151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.269929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm\") pod \"901ecc72-4eb3-49b8-be96-793614191b04\" (UID: \"901ecc72-4eb3-49b8-be96-793614191b04\") "
Sep 16 13:25:22.270151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.270102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home" (OuterVolumeSpecName: "home") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.270244 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.270117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache" (OuterVolumeSpecName: "model-cache") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.270244 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.270164    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.272103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.272069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:25:22.272216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.272107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm" (OuterVolumeSpecName: "dshm") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.272216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.272120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm" (OuterVolumeSpecName: "kube-api-access-cd4vm") pod "901ecc72-4eb3-49b8-be96-793614191b04" (UID: "901ecc72-4eb3-49b8-be96-793614191b04"). InnerVolumeSpecName "kube-api-access-cd4vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:25:22.370919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370859    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.370919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370912    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd4vm\" (UniqueName: \"kubernetes.io/projected/901ecc72-4eb3-49b8-be96-793614191b04-kube-api-access-cd4vm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.370919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370927    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.371180 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370939    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.371180 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370951    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/901ecc72-4eb3-49b8-be96-793614191b04-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.371180 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.370961    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/901ecc72-4eb3-49b8-be96-793614191b04-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.487985 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.487939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" podUID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" containerName="main" containerID="cri-o://d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c" gracePeriod=45
Sep 16 13:25:22.732122 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.732097    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:25:22.774275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774462 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bbns\" (UniqueName: \"kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns\") pod \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\" (UID: \"6052a105-eb0c-4ed4-bf0d-5159bd14dd01\") "
Sep 16 13:25:22.774705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home" (OuterVolumeSpecName: "home") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.774705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774658    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache" (OuterVolumeSpecName: "model-cache") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.774875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774821    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.774875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.774875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.774839    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.776519 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.776491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:25:22.776647 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.776621    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm" (OuterVolumeSpecName: "dshm") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:25:22.776647 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.776635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns" (OuterVolumeSpecName: "kube-api-access-9bbns") pod "6052a105-eb0c-4ed4-bf0d-5159bd14dd01" (UID: "6052a105-eb0c-4ed4-bf0d-5159bd14dd01"). InnerVolumeSpecName "kube-api-access-9bbns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:25:22.876185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.876145    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.876185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.876177    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.876185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.876186    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bbns\" (UniqueName: \"kubernetes.io/projected/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-kube-api-access-9bbns\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:22.876185 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:22.876196    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6052a105-eb0c-4ed4-bf0d-5159bd14dd01-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:25:23.051397 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.051310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"
Sep 16 13:25:23.051920 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.051316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9" event={"ID":"901ecc72-4eb3-49b8-be96-793614191b04","Type":"ContainerDied","Data":"818b79f4e023fce6031425f67b96af9494b6e11505878c3dd475cdb9b392830c"}
Sep 16 13:25:23.051920 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.051465    2579 scope.go:117] "RemoveContainer" containerID="b695da51f76fc57fe7694106cf5be16193f3c121069860944ee8fe70e3e5647f"
Sep 16 13:25:23.052909 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.052828    2579 generic.go:356] "Generic (PLEG): container finished" podID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" containerID="d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c" exitCode=0
Sep 16 13:25:23.052909 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.052862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"
Sep 16 13:25:23.052909 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.052904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" event={"ID":"6052a105-eb0c-4ed4-bf0d-5159bd14dd01","Type":"ContainerDied","Data":"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"}
Sep 16 13:25:23.053087 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.052942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx" event={"ID":"6052a105-eb0c-4ed4-bf0d-5159bd14dd01","Type":"ContainerDied","Data":"d1643826ed938270996211a787232b02aac977c29a7ce7a07e7f03540a0f4323"}
Sep 16 13:25:23.061614 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.061594    2579 scope.go:117] "RemoveContainer" containerID="d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"
Sep 16 13:25:23.069473 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.069454    2579 scope.go:117] "RemoveContainer" containerID="d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"
Sep 16 13:25:23.069812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.069789    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:25:23.069912 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:25:23.069797    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c\": container with ID starting with d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c not found: ID does not exist" containerID="d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"
Sep 16 13:25:23.069912 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.069844    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c"} err="failed to get container status \"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c\": rpc error: code = NotFound desc = could not find container \"d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c\": container with ID starting with d218ba82edcbc15fbc23e6709665d46361c714efb2d61dbc6bceaf48d42d863c not found: ID does not exist"
Sep 16 13:25:23.074654 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.074630    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-htps9"]
Sep 16 13:25:23.092879 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.092847    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:25:23.097348 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:23.097319    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-wclrx"]
Sep 16 13:25:24.463020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:24.462987    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" path="/var/lib/kubelet/pods/6052a105-eb0c-4ed4-bf0d-5159bd14dd01/volumes"
Sep 16 13:25:24.463394 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:24.463367    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901ecc72-4eb3-49b8-be96-793614191b04" path="/var/lib/kubelet/pods/901ecc72-4eb3-49b8-be96-793614191b04/volumes"
Sep 16 13:25:24.926491 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:24.926454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:25:28.292146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:25:28.292109    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv_23b097f5-efb4-44b1-8289-dc9ef93da2a5/main/0.log"
Sep 16 13:26:29.116471 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.116433    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:26:29.116956 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.116777    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="main" containerID="cri-o://7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058" gracePeriod=30
Sep 16 13:26:29.126425 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.126396    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:26:29.351331 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:29.351293    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:26:29.351331 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.351331 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.351331 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0" cmd=["/bin/sleep","15"]
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.351346    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" message="" err=<
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:29.351359    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main" containerID="cri-o://c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"
Sep 16 13:26:29.351646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.351405    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main" containerID="cri-o://c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0" gracePeriod=60
Sep 16 13:26:29.763739 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.763716    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:26:29.818836 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.818804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location\") pod \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") "
Sep 16 13:26:29.819021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.818868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir\") pod \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") "
Sep 16 13:26:29.819021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.818951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert\") pod \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") "
Sep 16 13:26:29.819021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.818987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs\") pod \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\" (UID: \"23b097f5-efb4-44b1-8289-dc9ef93da2a5\") "
Sep 16 13:26:29.819319 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.819279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23b097f5-efb4-44b1-8289-dc9ef93da2a5" (UID: "23b097f5-efb4-44b1-8289-dc9ef93da2a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.819319 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.819286    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23b097f5-efb4-44b1-8289-dc9ef93da2a5" (UID: "23b097f5-efb4-44b1-8289-dc9ef93da2a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:26:29.821235 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.821214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23b097f5-efb4-44b1-8289-dc9ef93da2a5" (UID: "23b097f5-efb4-44b1-8289-dc9ef93da2a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:29.840665 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.840627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23b097f5-efb4-44b1-8289-dc9ef93da2a5" (UID: "23b097f5-efb4-44b1-8289-dc9ef93da2a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.920509 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.920469    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b097f5-efb4-44b1-8289-dc9ef93da2a5-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:29.920509 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.920502    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:29.920509 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.920512    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:29.920779 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.920522    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b097f5-efb4-44b1-8289-dc9ef93da2a5-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:29.939679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:29.939640    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 16 13:26:30.083538 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.083511    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:26:30.223533 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsg2p\" (UniqueName: \"kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223621    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs\") pod \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\" (UID: \"3eb240ec-14fb-44bb-90ee-b6b696b2eefd\") "
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.223883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home" (OuterVolumeSpecName: "home") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.224057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.224013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.224532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.224120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.224532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.224140    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.224532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.224158    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.224532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.224275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:26:30.225911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.225884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm" (OuterVolumeSpecName: "dshm") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.225911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.225897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p" (OuterVolumeSpecName: "kube-api-access-bsg2p") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "kube-api-access-bsg2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:26:30.226052 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.225941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:30.246619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.246592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3eb240ec-14fb-44bb-90ee-b6b696b2eefd" (UID: "3eb240ec-14fb-44bb-90ee-b6b696b2eefd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325464    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325496    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg2p\" (UniqueName: \"kubernetes.io/projected/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kube-api-access-bsg2p\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325508    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325516    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325527    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.325556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.325536    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb240ec-14fb-44bb-90ee-b6b696b2eefd-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.348619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.348548    2579 generic.go:356] "Generic (PLEG): container finished" podID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerID="c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0" exitCode=0
Sep 16 13:26:30.348619 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.348619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"
Sep 16 13:26:30.348889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.348632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerDied","Data":"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"}
Sep 16 13:26:30.348889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.348692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths" event={"ID":"3eb240ec-14fb-44bb-90ee-b6b696b2eefd","Type":"ContainerDied","Data":"521765b023fcf36f98e6994b26c5906a6dd8fb40901f68c8540df7fa1237165a"}
Sep 16 13:26:30.348889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.348707    2579 scope.go:117] "RemoveContainer" containerID="c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"
Sep 16 13:26:30.350607 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.350580    2579 generic.go:356] "Generic (PLEG): container finished" podID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerID="7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058" exitCode=0
Sep 16 13:26:30.350727 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.350620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerDied","Data":"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"}
Sep 16 13:26:30.350727 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.350646    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"
Sep 16 13:26:30.350727 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.350663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv" event={"ID":"23b097f5-efb4-44b1-8289-dc9ef93da2a5","Type":"ContainerDied","Data":"03740092232d085a545e1e8f02b7893a902416b16dd7319a59a47c78105ccce6"}
Sep 16 13:26:30.359587 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.359371    2579 scope.go:117] "RemoveContainer" containerID="e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"
Sep 16 13:26:30.367606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.367586    2579 scope.go:117] "RemoveContainer" containerID="c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"
Sep 16 13:26:30.367927 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:30.367905    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0\": container with ID starting with c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0 not found: ID does not exist" containerID="c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"
Sep 16 13:26:30.368011 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.367940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0"} err="failed to get container status \"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0\": rpc error: code = NotFound desc = could not find container \"c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0\": container with ID starting with c4ee24851a93f16a54dc48ec2f89209b9e7af75bed3162716aee31034d6a3fb0 not found: ID does not exist"
Sep 16 13:26:30.368011 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.367975    2579 scope.go:117] "RemoveContainer" containerID="e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"
Sep 16 13:26:30.368276 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:30.368258    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927\": container with ID starting with e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927 not found: ID does not exist" containerID="e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"
Sep 16 13:26:30.368316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.368282    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927"} err="failed to get container status \"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927\": rpc error: code = NotFound desc = could not find container \"e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927\": container with ID starting with e041d2f2e3bf0f0d53799b90dc9736d3c21fcaee27813fefc1fd20ef446f3927 not found: ID does not exist"
Sep 16 13:26:30.368316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.368299    2579 scope.go:117] "RemoveContainer" containerID="7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"
Sep 16 13:26:30.375704 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.375681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:26:30.377304 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.377272    2579 scope.go:117] "RemoveContainer" containerID="f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"
Sep 16 13:26:30.380046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.380026    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8499b8cccf-c77mv"]
Sep 16 13:26:30.386355 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.386328    2579 scope.go:117] "RemoveContainer" containerID="7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"
Sep 16 13:26:30.386674 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:30.386646    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058\": container with ID starting with 7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058 not found: ID does not exist" containerID="7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"
Sep 16 13:26:30.386786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.386684    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058"} err="failed to get container status \"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058\": rpc error: code = NotFound desc = could not find container \"7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058\": container with ID starting with 7d5c4ea8def606576279014647c692481e174b2bd3b36323fde02b19eb090058 not found: ID does not exist"
Sep 16 13:26:30.386786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.386710    2579 scope.go:117] "RemoveContainer" containerID="f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"
Sep 16 13:26:30.387011 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:26:30.386993    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece\": container with ID starting with f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece not found: ID does not exist" containerID="f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"
Sep 16 13:26:30.387053 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.387018    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece"} err="failed to get container status \"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece\": rpc error: code = NotFound desc = could not find container \"f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece\": container with ID starting with f7189d59a38e957f4c8a73cec7a0784fc2c9bd48ab8bda157c8ccbf9f10efece not found: ID does not exist"
Sep 16 13:26:30.393838 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.393818    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:26:30.396039 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.396015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-7lths"]
Sep 16 13:26:30.471188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.471155    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" path="/var/lib/kubelet/pods/23b097f5-efb4-44b1-8289-dc9ef93da2a5/volumes"
Sep 16 13:26:30.471643 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:26:30.471627    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" path="/var/lib/kubelet/pods/3eb240ec-14fb-44bb-90ee-b6b696b2eefd/volumes"
Sep 16 13:27:13.999003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.998965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999252    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999262    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999273    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="storage-initializer"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="storage-initializer"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901ecc72-4eb3-49b8-be96-793614191b04" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999294    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="901ecc72-4eb3-49b8-be96-793614191b04" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999300    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999312    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999317    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999326    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="storage-initializer"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999330    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="storage-initializer"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999374    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6052a105-eb0c-4ed4-bf0d-5159bd14dd01" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999381    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb240ec-14fb-44bb-90ee-b6b696b2eefd" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999387    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b097f5-efb4-44b1-8289-dc9ef93da2a5" containerName="main"
Sep 16 13:27:13.999395 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:13.999393    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="901ecc72-4eb3-49b8-be96-793614191b04" containerName="main"
Sep 16 13:27:14.002232 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.002216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.006248 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.006218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 13:27:14.006248 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.006218    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 13:27:14.006505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.006269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 13:27:14.006505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.006268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-2pvkf\""
Sep 16 13:27:14.013579 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.013535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:27:14.094803 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.094803 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtsb\" (UniqueName: \"kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.095023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.095023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.095023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.095023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.094951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.195996 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.195955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtsb\" (UniqueName: \"kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196476 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196517 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.196645 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.196629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.198377 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.198361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.198746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.198729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.206497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.206472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtsb\" (UniqueName: \"kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb\") pod \"late-v1-kserve-56cbc8456c-8vp4r\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.312788 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.312753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:14.609726 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:14.609678    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:27:14.611990 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:27:14.611954    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50737a6b_ec73_428d_b1fc_6473c324533f.slice/crio-343b2dbbf6d0611c39e709477f5213cc130cbb7b73c6521ae0ee8918bdf8e770 WatchSource:0}: Error finding container 343b2dbbf6d0611c39e709477f5213cc130cbb7b73c6521ae0ee8918bdf8e770: Status 404 returned error can't find the container with id 343b2dbbf6d0611c39e709477f5213cc130cbb7b73c6521ae0ee8918bdf8e770
Sep 16 13:27:15.544670 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:15.544626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" event={"ID":"50737a6b-ec73-428d-b1fc-6473c324533f","Type":"ContainerStarted","Data":"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"}
Sep 16 13:27:15.545189 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:15.544682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" event={"ID":"50737a6b-ec73-428d-b1fc-6473c324533f","Type":"ContainerStarted","Data":"343b2dbbf6d0611c39e709477f5213cc130cbb7b73c6521ae0ee8918bdf8e770"}
Sep 16 13:27:15.567668 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:15.567604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" podStartSLOduration=2.5675820959999998 podStartE2EDuration="2.567582096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:15.566711197 +0000 UTC m=+2341.605510322" watchObservedRunningTime="2026-09-16 13:27:15.567582096 +0000 UTC m=+2341.606381217"
Sep 16 13:27:16.313792 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:16.313752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:24.313700 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:24.313666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:24.326275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:24.326246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:24.593587 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:24.593483    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:27:49.061986 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.061945    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:27:49.065238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.065220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.068217 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.068196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 13:27:49.068334 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.068283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 13:27:49.069382 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.069363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 13:27:49.069382 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.069376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:27:49.069536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.069379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 13:27:49.069536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.069384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-d4hkr\""
Sep 16 13:27:49.085645 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.085617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:27:49.183677 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.183865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.183865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgzb\" (UniqueName: \"kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.183865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.183865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.184018 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.184018 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.184018 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.183961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284453 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284453 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgzb\" (UniqueName: \"kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.284733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.285045 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.284963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.285045 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.285013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.285045 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.285041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.285199 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.285135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.285356 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.285331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.287025 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.286992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.287152 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.287134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.292470 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.292448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgzb\" (UniqueName: \"kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.347725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.347608    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:27:49.352247 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.352218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.364628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.364596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:27:49.376739 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.376704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:49.486274 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.486226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.486477 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.486319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.486477 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.486367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.486477 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.486448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.586890 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.586850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587104 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.586899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587104 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.586964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587104 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.587007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.587312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587429 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.587404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.587530 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.587498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.589664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.589644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.651375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.651344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:27:49.654345 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:27:49.654316    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc744f91e_ef96_4c64_94cf_f6a4e2aa0278.slice/crio-11fde7f5d6a6dddf84efde86cbb94dc1dcaea6c046db159eb9d1ed09b2f82c4f WatchSource:0}: Error finding container 11fde7f5d6a6dddf84efde86cbb94dc1dcaea6c046db159eb9d1ed09b2f82c4f: Status 404 returned error can't find the container with id 11fde7f5d6a6dddf84efde86cbb94dc1dcaea6c046db159eb9d1ed09b2f82c4f
Sep 16 13:27:49.664119 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.664100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:49.688063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.688031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerStarted","Data":"11fde7f5d6a6dddf84efde86cbb94dc1dcaea6c046db159eb9d1ed09b2f82c4f"}
Sep 16 13:27:49.941802 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:49.941775    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:27:49.944491 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:27:49.944462    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f1dfb7_86d7_44b4_be19_7f0bba39d3e6.slice/crio-4c58f62f1bb97360ce850f57b6a1a028ac97ec5b67165f8e107f8f334467b8fb WatchSource:0}: Error finding container 4c58f62f1bb97360ce850f57b6a1a028ac97ec5b67165f8e107f8f334467b8fb: Status 404 returned error can't find the container with id 4c58f62f1bb97360ce850f57b6a1a028ac97ec5b67165f8e107f8f334467b8fb
Sep 16 13:27:50.694311 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:50.694271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerStarted","Data":"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"}
Sep 16 13:27:50.696123 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:50.696090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerStarted","Data":"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"}
Sep 16 13:27:50.696245 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:50.696151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerStarted","Data":"4c58f62f1bb97360ce850f57b6a1a028ac97ec5b67165f8e107f8f334467b8fb"}
Sep 16 13:27:52.709736 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:52.709652    2579 generic.go:356] "Generic (PLEG): container finished" podID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerID="eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8" exitCode=0
Sep 16 13:27:52.710144 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:52.709727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerDied","Data":"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"}
Sep 16 13:27:52.711328 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:52.711309    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerID="8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c" exitCode=0
Sep 16 13:27:52.711413 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:52.711381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerDied","Data":"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"}
Sep 16 13:27:53.718006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:53.717967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerStarted","Data":"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"}
Sep 16 13:27:53.720070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:53.720045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerStarted","Data":"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"}
Sep 16 13:27:53.720254 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:53.720237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:27:53.746139 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:53.746089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" podStartSLOduration=2.081014815 podStartE2EDuration="4.746073286s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.665058471s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:53.744763642 +0000 UTC m=+2379.783562806" watchObservedRunningTime="2026-09-16 13:27:53.746073286 +0000 UTC m=+2379.784872409"
Sep 16 13:27:53.764045 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:53.763996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" podStartSLOduration=2.181560094 podStartE2EDuration="4.763981878s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.582421784s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:53.763722115 +0000 UTC m=+2379.802521239" watchObservedRunningTime="2026-09-16 13:27:53.763981878 +0000 UTC m=+2379.802780998"
Sep 16 13:27:54.376924 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:54.376877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:59.377448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:59.377405    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:59.390213 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:59.390180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:27:59.758172 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:27:59.758135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:28:14.515342 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:14.515313    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:28:14.520228 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:14.520206    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:28:24.735495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:24.735459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:28:35.631858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:35.631822    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb_c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6/main/0.log"
Sep 16 13:28:42.535970 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.535870    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:28:42.540821 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.540796    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:28:42.541042 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.541004    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="main" containerID="cri-o://e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5" gracePeriod=30
Sep 16 13:28:42.545906 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:42.545876    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:28:42.545906 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.545906 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.545906 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4" cmd=["/bin/sleep","15"]
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.545927    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" message="" err=<
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:42.545938    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main" containerID="cri-o://0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"
Sep 16 13:28:42.546125 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.545987    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main" containerID="cri-o://0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4" gracePeriod=60
Sep 16 13:28:42.747207 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.747165    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 16 13:28:42.895648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.895624    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:28:42.931667 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.931635    2579 generic.go:356] "Generic (PLEG): container finished" podID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerID="0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4" exitCode=0
Sep 16 13:28:42.931833 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.931694    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"
Sep 16 13:28:42.931833 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.931716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerDied","Data":"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"}
Sep 16 13:28:42.931833 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.931799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm" event={"ID":"c744f91e-ef96-4c64-94cf-f6a4e2aa0278","Type":"ContainerDied","Data":"11fde7f5d6a6dddf84efde86cbb94dc1dcaea6c046db159eb9d1ed09b2f82c4f"}
Sep 16 13:28:42.931949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.931834    2579 scope.go:117] "RemoveContainer" containerID="0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"
Sep 16 13:28:42.941290 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.941269    2579 scope.go:117] "RemoveContainer" containerID="eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"
Sep 16 13:28:42.949903 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.949882    2579 scope.go:117] "RemoveContainer" containerID="0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"
Sep 16 13:28:42.950185 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:42.950166    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4\": container with ID starting with 0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4 not found: ID does not exist" containerID="0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"
Sep 16 13:28:42.950261 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.950192    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4"} err="failed to get container status \"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4\": rpc error: code = NotFound desc = could not find container \"0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4\": container with ID starting with 0b42c32614c873b4541c98b7144880bc4dc823901a011497a6a476546ca19ce4 not found: ID does not exist"
Sep 16 13:28:42.950261 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.950210    2579 scope.go:117] "RemoveContainer" containerID="eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"
Sep 16 13:28:42.950525 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:42.950498    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8\": container with ID starting with eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8 not found: ID does not exist" containerID="eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"
Sep 16 13:28:42.950669 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:42.950532    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8"} err="failed to get container status \"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8\": rpc error: code = NotFound desc = could not find container \"eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8\": container with ID starting with eeb02eaff3db9e4b7c55296973692f86c704245a143b486ace542e707851e9f8 not found: ID does not exist"
Sep 16 13:28:43.039429 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039550 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039550 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039550 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lgzb\" (UniqueName: \"kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert\") pod \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\" (UID: \"c744f91e-ef96-4c64-94cf-f6a4e2aa0278\") "
Sep 16 13:28:43.039913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039707    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache" (OuterVolumeSpecName: "model-cache") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.039913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home" (OuterVolumeSpecName: "home") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.040034 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039931    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.040034 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039947    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.040034 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.039976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.040163 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.040088    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:28:43.041972 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.041952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:28:43.042071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.042026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb" (OuterVolumeSpecName: "kube-api-access-5lgzb") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "kube-api-access-5lgzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:28:43.042126 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.042107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm" (OuterVolumeSpecName: "dshm") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.062439 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.062415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c744f91e-ef96-4c64-94cf-f6a4e2aa0278" (UID: "c744f91e-ef96-4c64-94cf-f6a4e2aa0278"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.141093 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141062    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.141093 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgzb\" (UniqueName: \"kubernetes.io/projected/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-kube-api-access-5lgzb\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.141277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141102    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.141277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141111    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.141277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141119    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.141277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.141127    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c744f91e-ef96-4c64-94cf-f6a4e2aa0278-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.185216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.185189    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:28:43.259270 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.259236    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:28:43.262510 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.262484    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-s8cbm"]
Sep 16 13:28:43.342610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.342499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs\") pod \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") "
Sep 16 13:28:43.342610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.342553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location\") pod \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") "
Sep 16 13:28:43.342784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.342633    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert\") pod \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") "
Sep 16 13:28:43.342784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.342665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir\") pod \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\" (UID: \"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6\") "
Sep 16 13:28:43.343039 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.343013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" (UID: "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:28:43.343153 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.343095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" (UID: "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.344713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.344694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" (UID: "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:28:43.363578 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.363527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" (UID: "c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.443883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.443851    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.443883 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.443879    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.444055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.443891    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.444055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.443899    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.938449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.938355    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerID="e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5" exitCode=0
Sep 16 13:28:43.938449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.938399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerDied","Data":"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"}
Sep 16 13:28:43.938449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.938415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"
Sep 16 13:28:43.939013 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.938452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb" event={"ID":"c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6","Type":"ContainerDied","Data":"4c58f62f1bb97360ce850f57b6a1a028ac97ec5b67165f8e107f8f334467b8fb"}
Sep 16 13:28:43.939013 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.938483    2579 scope.go:117] "RemoveContainer" containerID="e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"
Sep 16 13:28:43.947002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.946982    2579 scope.go:117] "RemoveContainer" containerID="8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"
Sep 16 13:28:43.954403 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.954385    2579 scope.go:117] "RemoveContainer" containerID="e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"
Sep 16 13:28:43.954675 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:43.954654    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5\": container with ID starting with e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5 not found: ID does not exist" containerID="e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"
Sep 16 13:28:43.954783 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.954680    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5"} err="failed to get container status \"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5\": rpc error: code = NotFound desc = could not find container \"e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5\": container with ID starting with e2cd343f110b2ecc4eb60f9a9a63ff6634ad637a278a605a15d6e7c00c065ae5 not found: ID does not exist"
Sep 16 13:28:43.954783 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.954697    2579 scope.go:117] "RemoveContainer" containerID="8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"
Sep 16 13:28:43.954947 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:28:43.954934    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c\": container with ID starting with 8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c not found: ID does not exist" containerID="8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"
Sep 16 13:28:43.954995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.954949    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c"} err="failed to get container status \"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c\": rpc error: code = NotFound desc = could not find container \"8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c\": container with ID starting with 8382d3255b9a962ac60fe34e28682e4c750f603ec50a6612dbf477583faa059c not found: ID does not exist"
Sep 16 13:28:43.959904 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.959882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:28:43.965423 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:43.965398    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7cd97888d7-vprwb"]
Sep 16 13:28:44.458643 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:44.458609    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" path="/var/lib/kubelet/pods/c744f91e-ef96-4c64-94cf-f6a4e2aa0278/volumes"
Sep 16 13:28:44.459041 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:28:44.459028    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" path="/var/lib/kubelet/pods/c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6/volumes"
Sep 16 13:29:10.748808 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:10.748763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:29:22.799792 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.799759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800176    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="storage-initializer"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800194    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="storage-initializer"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800211    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="main"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="main"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800234    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="storage-initializer"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800242    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="storage-initializer"
Sep 16 13:29:22.800257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main"
Sep 16 13:29:22.800637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800263    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main"
Sep 16 13:29:22.800637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f1dfb7-86d7-44b4-be19-7f0bba39d3e6" containerName="main"
Sep 16 13:29:22.800637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.800352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c744f91e-ef96-4c64-94cf-f6a4e2aa0278" containerName="main"
Sep 16 13:29:22.803251 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.803232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.806146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.806127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-5rrfm\""
Sep 16 13:29:22.807223 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.807204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 13:29:22.807223 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.807216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 13:29:22.807528 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.807510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 13:29:22.813584 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.813545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:29:22.859658 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.859788 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.859788 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.859788 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.859892 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9j6\" (UniqueName: \"kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.859892 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.859871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.960948 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.960919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9j6\" (UniqueName: \"kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.961221 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.961186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.961296 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.961273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.961345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.961309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.963229 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.963205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.963421 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.963406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:22.968790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:22.968768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9j6\" (UniqueName: \"kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6\") pod \"delw-v1-kserve-6cf8cbb847-d7xns\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:23.113833 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:23.113751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:23.393150 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:23.393054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:29:23.397064 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:29:23.397025    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac52f3f_5f3d_48c6_b4ba_9c4f17e1a851.slice/crio-f36c56177ea02ed6527725451e590793c887dd520204c95a24e0248262e30a41 WatchSource:0}: Error finding container f36c56177ea02ed6527725451e590793c887dd520204c95a24e0248262e30a41: Status 404 returned error can't find the container with id f36c56177ea02ed6527725451e590793c887dd520204c95a24e0248262e30a41
Sep 16 13:29:24.111421 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:24.111382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" event={"ID":"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851","Type":"ContainerStarted","Data":"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"}
Sep 16 13:29:24.111817 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:24.111443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" event={"ID":"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851","Type":"ContainerStarted","Data":"f36c56177ea02ed6527725451e590793c887dd520204c95a24e0248262e30a41"}
Sep 16 13:29:24.114686 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:24.114667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:24.129956 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:24.129904    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" podStartSLOduration=2.129889532 podStartE2EDuration="2.129889532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:29:24.129360073 +0000 UTC m=+2470.168159196" watchObservedRunningTime="2026-09-16 13:29:24.129889532 +0000 UTC m=+2470.168688654"
Sep 16 13:29:25.768241 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:25.768198    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" podUID="50737a6b-ec73-428d-b1fc-6473c324533f" containerName="main" containerID="cri-o://a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f" gracePeriod=45
Sep 16 13:29:26.016879 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.016858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:29:26.090914 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.090835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.090914 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.090869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.091103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.090917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.091103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.090940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.091103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.090986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjtsb\" (UniqueName: \"kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.091103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.091020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir\") pod \"50737a6b-ec73-428d-b1fc-6473c324533f\" (UID: \"50737a6b-ec73-428d-b1fc-6473c324533f\") "
Sep 16 13:29:26.091328 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.091214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache" (OuterVolumeSpecName: "model-cache") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:29:26.091328 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.091226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home" (OuterVolumeSpecName: "home") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:29:26.091432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.091411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:29:26.093208 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.093186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb" (OuterVolumeSpecName: "kube-api-access-qjtsb") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "kube-api-access-qjtsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:29:26.093474 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.093452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm" (OuterVolumeSpecName: "dshm") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:29:26.093518 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.093452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50737a6b-ec73-428d-b1fc-6473c324533f" (UID: "50737a6b-ec73-428d-b1fc-6473c324533f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:29:26.121079 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.121049    2579 generic.go:356] "Generic (PLEG): container finished" podID="50737a6b-ec73-428d-b1fc-6473c324533f" containerID="a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f" exitCode=0
Sep 16 13:29:26.121190 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.121098    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"
Sep 16 13:29:26.121190 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.121136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" event={"ID":"50737a6b-ec73-428d-b1fc-6473c324533f","Type":"ContainerDied","Data":"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"}
Sep 16 13:29:26.121190 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.121187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r" event={"ID":"50737a6b-ec73-428d-b1fc-6473c324533f","Type":"ContainerDied","Data":"343b2dbbf6d0611c39e709477f5213cc130cbb7b73c6521ae0ee8918bdf8e770"}
Sep 16 13:29:26.121309 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.121210    2579 scope.go:117] "RemoveContainer" containerID="a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"
Sep 16 13:29:26.131656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.131633    2579 scope.go:117] "RemoveContainer" containerID="a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"
Sep 16 13:29:26.132084 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:29:26.131949    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f\": container with ID starting with a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f not found: ID does not exist" containerID="a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"
Sep 16 13:29:26.132084 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.131988    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f"} err="failed to get container status \"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f\": rpc error: code = NotFound desc = could not find container \"a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f\": container with ID starting with a1d2cc76b122b4cfd49ae87de64d9dfc13754feb95fb1aa7848eeb8915b8788f not found: ID does not exist"
Sep 16 13:29:26.147293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.147269    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:29:26.153702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.153680    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-8vp4r"]
Sep 16 13:29:26.192540 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192509    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.192540 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192535    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.192789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192548    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.192789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192573    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50737a6b-ec73-428d-b1fc-6473c324533f-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.192789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192583    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjtsb\" (UniqueName: \"kubernetes.io/projected/50737a6b-ec73-428d-b1fc-6473c324533f-kube-api-access-qjtsb\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.192789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.192591    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50737a6b-ec73-428d-b1fc-6473c324533f-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:29:26.457371 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:26.457337    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50737a6b-ec73-428d-b1fc-6473c324533f" path="/var/lib/kubelet/pods/50737a6b-ec73-428d-b1fc-6473c324533f/volumes"
Sep 16 13:29:30.390461 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.390425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:29:30.390837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.390726    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50737a6b-ec73-428d-b1fc-6473c324533f" containerName="main"
Sep 16 13:29:30.390837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.390737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50737a6b-ec73-428d-b1fc-6473c324533f" containerName="main"
Sep 16 13:29:30.390837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.390793    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50737a6b-ec73-428d-b1fc-6473c324533f" containerName="main"
Sep 16 13:29:30.394838 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.394818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.397531 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.397510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 13:29:30.407226 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.407206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:29:30.529537 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.529537 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.529815 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.529815 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sk9\" (UniqueName: \"kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.529815 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.529815 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.529724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630599 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sk9\" (UniqueName: \"kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.630750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.630713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.631029 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.631005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.631103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.631027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.631192 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.631162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.632989 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.632962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.633239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.633220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.639030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.639007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sk9\" (UniqueName: \"kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9\") pod \"delw-v2-kserve-5b84cc7ddf-x582j\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.704555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.704527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:30.979981 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:30.979950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:29:30.982248 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:29:30.982216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b14b2b4_6ebe_47fc_a6e6_7946f74cdae7.slice/crio-50ed391a22bf6387da1e1959ea152b8ae5e9056e4a049ec5e9e4788d4091614f WatchSource:0}: Error finding container 50ed391a22bf6387da1e1959ea152b8ae5e9056e4a049ec5e9e4788d4091614f: Status 404 returned error can't find the container with id 50ed391a22bf6387da1e1959ea152b8ae5e9056e4a049ec5e9e4788d4091614f
Sep 16 13:29:31.147828 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:31.147781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" event={"ID":"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7","Type":"ContainerStarted","Data":"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"}
Sep 16 13:29:31.148041 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:31.147849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" event={"ID":"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7","Type":"ContainerStarted","Data":"50ed391a22bf6387da1e1959ea152b8ae5e9056e4a049ec5e9e4788d4091614f"}
Sep 16 13:29:31.169621 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:31.169543    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" podStartSLOduration=1.169524266 podStartE2EDuration="1.169524266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:29:31.167937057 +0000 UTC m=+2477.206736195" watchObservedRunningTime="2026-09-16 13:29:31.169524266 +0000 UTC m=+2477.208323389"
Sep 16 13:29:31.705306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:31.705264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:33.114614 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:33.114573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:33.127293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:33.127266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:33.167329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:33.167300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:29:40.704748 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:40.704710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:40.717016 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:40.716988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:29:41.202163 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:29:41.202130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:30:15.045863 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.045776    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:30:15.050200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.050178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.054238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.054210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 13:30:15.054238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.054223    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 13:30:15.054449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.054251    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 13:30:15.054449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.054257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-qt9sm\""
Sep 16 13:30:15.054449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.054371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-mvs8j\""
Sep 16 13:30:15.059034 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.059013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:30:15.106944 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.106901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55lz\" (UniqueName: \"kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.107161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.106985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.208061 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.208025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.208246 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.208109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55lz\" (UniqueName: \"kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.210576 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.210537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.218134 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.218103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55lz\" (UniqueName: \"kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.361130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.361043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:15.636771 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:15.636682    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:30:15.639800 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:30:15.639766    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc196b3_4bc4_4dca_946e_b17b2109a67b.slice/crio-a7b98875bbc1ef75be0f69b9464d27bc133188cf280ff8657bc447e756d7b5bb WatchSource:0}: Error finding container a7b98875bbc1ef75be0f69b9464d27bc133188cf280ff8657bc447e756d7b5bb: Status 404 returned error can't find the container with id a7b98875bbc1ef75be0f69b9464d27bc133188cf280ff8657bc447e756d7b5bb
Sep 16 13:30:16.348457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:16.348416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" event={"ID":"3fc196b3-4bc4-4dca-946e-b17b2109a67b","Type":"ContainerStarted","Data":"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"}
Sep 16 13:30:16.348938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:16.348478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" event={"ID":"3fc196b3-4bc4-4dca-946e-b17b2109a67b","Type":"ContainerStarted","Data":"a7b98875bbc1ef75be0f69b9464d27bc133188cf280ff8657bc447e756d7b5bb"}
Sep 16 13:30:16.348938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:16.348550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:16.368242 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:16.368188    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podStartSLOduration=1.368170525 podStartE2EDuration="1.368170525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:16.367042667 +0000 UTC m=+2522.405841792" watchObservedRunningTime="2026-09-16 13:30:16.368170525 +0000 UTC m=+2522.406969672"
Sep 16 13:30:25.066750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:25.066713    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:30:40.086571 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.086513    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerName="main" containerID="cri-o://03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37" gracePeriod=45
Sep 16 13:30:40.191966 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.191920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 16 13:30:40.333607 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.333582    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:30:40.419479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8sk9\" (UniqueName: \"kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.419694 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.419694 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.419694 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.419877 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.419877 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home\") pod \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\" (UID: \"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7\") "
Sep 16 13:30:40.420011 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.419983    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:40.420087 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.420035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:40.420087 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.420058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home" (OuterVolumeSpecName: "home") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:40.421886 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.421859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm" (OuterVolumeSpecName: "dshm") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:40.421972 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.421936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9" (OuterVolumeSpecName: "kube-api-access-g8sk9") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "kube-api-access-g8sk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:30:40.422014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.421976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" (UID: "0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:30:40.454205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.454176    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerID="03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37" exitCode=0
Sep 16 13:30:40.454349 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.454291    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"
Sep 16 13:30:40.458541 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.458503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" event={"ID":"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7","Type":"ContainerDied","Data":"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"}
Sep 16 13:30:40.458713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.458553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j" event={"ID":"0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7","Type":"ContainerDied","Data":"50ed391a22bf6387da1e1959ea152b8ae5e9056e4a049ec5e9e4788d4091614f"}
Sep 16 13:30:40.458713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.458598    2579 scope.go:117] "RemoveContainer" containerID="03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"
Sep 16 13:30:40.467716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.467699    2579 scope.go:117] "RemoveContainer" containerID="03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"
Sep 16 13:30:40.468034 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:30:40.468009    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37\": container with ID starting with 03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37 not found: ID does not exist" containerID="03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"
Sep 16 13:30:40.468092 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.468043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37"} err="failed to get container status \"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37\": rpc error: code = NotFound desc = could not find container \"03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37\": container with ID starting with 03bbfbc5a71f8518c2ec0df647398903873712e6456912f14e92236733b28c37 not found: ID does not exist"
Sep 16 13:30:40.478469 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.478444    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:30:40.482576 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.482538    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-x582j"]
Sep 16 13:30:40.520375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520342    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.520375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520370    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.520585 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520383    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8sk9\" (UniqueName: \"kubernetes.io/projected/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-kube-api-access-g8sk9\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.520585 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520397    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.520585 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520410    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.520585 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.520420    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:40.734448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:40.734370    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:30:42.463315 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:42.463280    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" path="/var/lib/kubelet/pods/0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7/volumes"
Sep 16 13:30:47.354872 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:47.354839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:30:51.591865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.591820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:30:51.592350 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.592145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerName="main"
Sep 16 13:30:51.592350 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.592157    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerName="main"
Sep 16 13:30:51.592350 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.592215    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b14b2b4-6ebe-47fc-a6e6-7946f74cdae7" containerName="main"
Sep 16 13:30:51.596313 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.596291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.599632 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.599587    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 13:30:51.600453 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.600433    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 13:30:51.600770 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.600751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-t7pfh\""
Sep 16 13:30:51.600846 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.600775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 13:30:51.609902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.609877    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:30:51.708112 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.708297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.708297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22jm\" (UniqueName: \"kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.708297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.708297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.708464 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.708325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809179 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22jm\" (UniqueName: \"kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809819 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.809858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.809796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.811622 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.811597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.811746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.811728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.818268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.818244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22jm\" (UniqueName: \"kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm\") pod \"roll-v1-kserve-69cf8bcfc-gsmpm\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:51.906284 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:51.906250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:52.183802 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:52.183765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:30:52.187147 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:30:52.187108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9b58dc_a5c6_411d_bc22_c4d35ad96aab.slice/crio-8152cd64edf414cc34f7c74a9f9face4a4bd545096eb6a058cec0681a5eb61db WatchSource:0}: Error finding container 8152cd64edf414cc34f7c74a9f9face4a4bd545096eb6a058cec0681a5eb61db: Status 404 returned error can't find the container with id 8152cd64edf414cc34f7c74a9f9face4a4bd545096eb6a058cec0681a5eb61db
Sep 16 13:30:52.507216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:52.507126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" event={"ID":"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab","Type":"ContainerStarted","Data":"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"}
Sep 16 13:30:52.507216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:52.507181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" event={"ID":"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab","Type":"ContainerStarted","Data":"8152cd64edf414cc34f7c74a9f9face4a4bd545096eb6a058cec0681a5eb61db"}
Sep 16 13:30:52.533468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:52.533396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" podStartSLOduration=1.533373391 podStartE2EDuration="1.533373391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:52.5304073 +0000 UTC m=+2558.569206424" watchObservedRunningTime="2026-09-16 13:30:52.533373391 +0000 UTC m=+2558.572172514"
Sep 16 13:30:52.906433 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:52.906398    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:30:55.753007 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:55.752967    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" podUID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" containerName="main" containerID="cri-o://441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662" gracePeriod=45
Sep 16 13:30:56.010287 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.010219    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:30:56.150741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.150741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.151015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj9j6\" (UniqueName: \"kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.151015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.151015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.151015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.150871    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home\") pod \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\" (UID: \"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851\") "
Sep 16 13:30:56.151200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.151038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:56.151200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.151051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:56.151200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.151155    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.151362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.151206    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.151362 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.151241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home" (OuterVolumeSpecName: "home") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:56.153105 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.153072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:30:56.153226 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.153156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm" (OuterVolumeSpecName: "dshm") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:30:56.153308 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.153287    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6" (OuterVolumeSpecName: "kube-api-access-tj9j6") pod "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" (UID: "9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851"). InnerVolumeSpecName "kube-api-access-tj9j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:30:56.252466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.252429    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9j6\" (UniqueName: \"kubernetes.io/projected/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-kube-api-access-tj9j6\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.252466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.252460    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.252466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.252469    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.252466 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.252477    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:30:56.525989 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.525949    2579 generic.go:356] "Generic (PLEG): container finished" podID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" containerID="441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662" exitCode=0
Sep 16 13:30:56.526158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.525995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"
Sep 16 13:30:56.526158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.526027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" event={"ID":"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851","Type":"ContainerDied","Data":"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"}
Sep 16 13:30:56.526158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.526073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns" event={"ID":"9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851","Type":"ContainerDied","Data":"f36c56177ea02ed6527725451e590793c887dd520204c95a24e0248262e30a41"}
Sep 16 13:30:56.526158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.526099    2579 scope.go:117] "RemoveContainer" containerID="441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"
Sep 16 13:30:56.534331 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.534309    2579 scope.go:117] "RemoveContainer" containerID="441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"
Sep 16 13:30:56.534641 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:30:56.534618    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662\": container with ID starting with 441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662 not found: ID does not exist" containerID="441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"
Sep 16 13:30:56.534756 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.534652    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662"} err="failed to get container status \"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662\": rpc error: code = NotFound desc = could not find container \"441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662\": container with ID starting with 441a9947a6c99fd96c9b36077d86e172478a1ef02fbb0a7833cb67b4b5c2d662 not found: ID does not exist"
Sep 16 13:30:56.543847 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.543822    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:30:56.547309 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:56.547287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d7xns"]
Sep 16 13:30:58.457928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:30:58.457894    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" path="/var/lib/kubelet/pods/9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851/volumes"
Sep 16 13:31:01.907373 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:01.907336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:31:01.919646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:01.919619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:31:02.563673 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:02.563644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:31:06.591800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.591767    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:31:06.592177 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.592161    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" containerName="main"
Sep 16 13:31:06.592177 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.592177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" containerName="main"
Sep 16 13:31:06.592268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.592246    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac52f3f-5f3d-48c6-b4ba-9c4f17e1a851" containerName="main"
Sep 16 13:31:06.596405 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.596389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.599114 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.599090    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 13:31:06.605243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.605217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:31:06.635760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.635913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.635913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.635913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.635913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.636071 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.635916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968n8\" (UniqueName: \"kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.736837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.736966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968n8\" (UniqueName: \"kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737305 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.737259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737359 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.737316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.737359 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.737340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.739230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.739204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.739474 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.739457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.745437 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.745417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968n8\" (UniqueName: \"kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8\") pod \"roll-v2-kserve-d4dbf9d67-vzxbj\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:06.906644 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:06.906618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:07.177020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:07.176993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:31:07.179872 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:31:07.179836    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c7c1b6_c527_45d0_bedc_a2df8ac70f26.slice/crio-a057ece7bc188f5158011a202affae3a0bf3fef1f7a75fed5c3fe85a71042ac9 WatchSource:0}: Error finding container a057ece7bc188f5158011a202affae3a0bf3fef1f7a75fed5c3fe85a71042ac9: Status 404 returned error can't find the container with id a057ece7bc188f5158011a202affae3a0bf3fef1f7a75fed5c3fe85a71042ac9
Sep 16 13:31:07.575098 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:07.575002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" event={"ID":"60c7c1b6-c527-45d0-bedc-a2df8ac70f26","Type":"ContainerStarted","Data":"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"}
Sep 16 13:31:07.575098 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:07.575062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" event={"ID":"60c7c1b6-c527-45d0-bedc-a2df8ac70f26","Type":"ContainerStarted","Data":"a057ece7bc188f5158011a202affae3a0bf3fef1f7a75fed5c3fe85a71042ac9"}
Sep 16 13:31:07.596583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:07.596501    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" podStartSLOduration=1.596479459 podStartE2EDuration="1.596479459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:07.594941618 +0000 UTC m=+2573.633740744" watchObservedRunningTime="2026-09-16 13:31:07.596479459 +0000 UTC m=+2573.635278583"
Sep 16 13:31:07.907369 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:07.907338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:16.906740 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:16.906700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:16.919404 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:16.919368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:17.636295 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:17.636264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:31:43.360302 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:43.360211    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:31:47.354804 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:47.354760    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:31:57.354240 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:57.354204    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:31:58.383745 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:31:58.383682    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" containerID="cri-o://1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562" gracePeriod=45
Sep 16 13:32:07.354909 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:07.354865    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:32:07.355321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:07.354974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:32:17.354978 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:17.354936    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:32:27.354298 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:27.354260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:32:28.635852 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.635828    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:32:28.719856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.719818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55lz\" (UniqueName: \"kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz\") pod \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") "
Sep 16 13:32:28.720040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.719883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs\") pod \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\" (UID: \"3fc196b3-4bc4-4dca-946e-b17b2109a67b\") "
Sep 16 13:32:28.722206 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.722175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fc196b3-4bc4-4dca-946e-b17b2109a67b" (UID: "3fc196b3-4bc4-4dca-946e-b17b2109a67b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:32:28.722206 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.722183    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz" (OuterVolumeSpecName: "kube-api-access-w55lz") pod "3fc196b3-4bc4-4dca-946e-b17b2109a67b" (UID: "3fc196b3-4bc4-4dca-946e-b17b2109a67b"). InnerVolumeSpecName "kube-api-access-w55lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:32:28.820789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.820756    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc196b3-4bc4-4dca-946e-b17b2109a67b-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:28.820789 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.820786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55lz\" (UniqueName: \"kubernetes.io/projected/3fc196b3-4bc4-4dca-946e-b17b2109a67b-kube-api-access-w55lz\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:28.919877 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.919846    2579 generic.go:356] "Generic (PLEG): container finished" podID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerID="1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562" exitCode=0
Sep 16 13:32:28.920046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.919889    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"
Sep 16 13:32:28.920046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.919936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" event={"ID":"3fc196b3-4bc4-4dca-946e-b17b2109a67b","Type":"ContainerDied","Data":"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"}
Sep 16 13:32:28.920046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.919997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7" event={"ID":"3fc196b3-4bc4-4dca-946e-b17b2109a67b","Type":"ContainerDied","Data":"a7b98875bbc1ef75be0f69b9464d27bc133188cf280ff8657bc447e756d7b5bb"}
Sep 16 13:32:28.920046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.920025    2579 scope.go:117] "RemoveContainer" containerID="1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"
Sep 16 13:32:28.928886 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.928870    2579 scope.go:117] "RemoveContainer" containerID="1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"
Sep 16 13:32:28.929156 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:32:28.929134    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562\": container with ID starting with 1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562 not found: ID does not exist" containerID="1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"
Sep 16 13:32:28.929220 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.929167    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562"} err="failed to get container status \"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562\": rpc error: code = NotFound desc = could not find container \"1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562\": container with ID starting with 1b8a7d9f91939b9812c5091a887d5baf8a7f275b1f77fc96a35e3b71ea4f4562 not found: ID does not exist"
Sep 16 13:32:28.941798 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.941775    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:32:28.947751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:28.947729    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched65j7"]
Sep 16 13:32:30.457459 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:30.457427    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" path="/var/lib/kubelet/pods/3fc196b3-4bc4-4dca-946e-b17b2109a67b/volumes"
Sep 16 13:32:36.157210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:36.157171    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:32:36.162151 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:36.162125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:32:42.289204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.289170    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:32:42.289583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.289480    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main"
Sep 16 13:32:42.289583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.289492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main"
Sep 16 13:32:42.289583 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.289556    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc196b3-4bc4-4dca-946e-b17b2109a67b" containerName="main"
Sep 16 13:32:42.293821 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.293803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.297633 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.297607    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 13:32:42.297633 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.297634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 13:32:42.297837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.297634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-mrvpn\""
Sep 16 13:32:42.297837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.297620    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 13:32:42.297837 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.297704    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-kp5mb\""
Sep 16 13:32:42.302530 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.302507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:32:42.433621 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.433555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xx6\" (UniqueName: \"kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.433799 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.433700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.534449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.534405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.534673 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.534645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xx6\" (UniqueName: \"kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.537133 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.537106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.543126 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.543066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xx6\" (UniqueName: \"kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.604036 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.604002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:42.876657 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.876628    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:32:42.879242 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:32:42.879213    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a875713_0e4d_4962_b8e0_b13181a81282.slice/crio-dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2 WatchSource:0}: Error finding container dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2: Status 404 returned error can't find the container with id dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2
Sep 16 13:32:42.985048 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.985008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" event={"ID":"6a875713-0e4d-4962-b8e0-b13181a81282","Type":"ContainerStarted","Data":"ed49bc83108918f8dbb0bd5341e2d6e9107c9c1354ce7fb4310909f9c719b371"}
Sep 16 13:32:42.985227 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.985076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" event={"ID":"6a875713-0e4d-4962-b8e0-b13181a81282","Type":"ContainerStarted","Data":"dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2"}
Sep 16 13:32:42.985227 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:42.985146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:32:43.005506 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:43.005458    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podStartSLOduration=1.005443333 podStartE2EDuration="1.005443333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:43.004680017 +0000 UTC m=+2669.043479140" watchObservedRunningTime="2026-09-16 13:32:43.005443333 +0000 UTC m=+2669.044242492"
Sep 16 13:32:51.176530 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.176480    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" podUID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" containerName="main" containerID="cri-o://a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45" gracePeriod=45
Sep 16 13:32:51.180167 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.180106    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" podUID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" containerName="main" containerID="cri-o://87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be" gracePeriod=45
Sep 16 13:32:51.448823 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.448793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:32:51.451516 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.451495    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:32:51.611606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.611606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611741    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-968n8\" (UniqueName: \"kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c22jm\" (UniqueName: \"kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.611868 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home" (OuterVolumeSpecName: "home") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.611968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache\") pod \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\" (UID: \"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab\") "
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache\") pod \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\" (UID: \"60c7c1b6-c527-45d0-bedc-a2df8ac70f26\") "
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home" (OuterVolumeSpecName: "home") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache" (OuterVolumeSpecName: "model-cache") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612450    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612473    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612487    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612499    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.612703 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.612512    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.614181 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm" (OuterVolumeSpecName: "dshm") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.614291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:32:51.614433 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm" (OuterVolumeSpecName: "kube-api-access-c22jm") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "kube-api-access-c22jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:32:51.614493 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8" (OuterVolumeSpecName: "kube-api-access-968n8") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "kube-api-access-968n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:32:51.614493 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm" (OuterVolumeSpecName: "dshm") pod "60c7c1b6-c527-45d0-bedc-a2df8ac70f26" (UID: "60c7c1b6-c527-45d0-bedc-a2df8ac70f26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:32:51.614617 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.614535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" (UID: "5a9b58dc-a5c6-411d-bc22-c4d35ad96aab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:32:51.679454 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679415    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:32:51.679742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" containerName="main"
Sep 16 13:32:51.679796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" containerName="main"
Sep 16 13:32:51.679796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679765    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" containerName="main"
Sep 16 13:32:51.679796 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679773    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" containerName="main"
Sep 16 13:32:51.679897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679829    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" containerName="main"
Sep 16 13:32:51.679897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.679839    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" containerName="main"
Sep 16 13:32:51.683771 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.683751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.686434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.686355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-gkk2b\""
Sep 16 13:32:51.686434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.686393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 13:32:51.686434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.686397    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 13:32:51.686727 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.686466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 13:32:51.694489 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.694462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:32:51.713091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713058    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c22jm\" (UniqueName: \"kubernetes.io/projected/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-kube-api-access-c22jm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713086    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713097    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713107    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713115    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713122    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.713262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.713132    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968n8\" (UniqueName: \"kubernetes.io/projected/60c7c1b6-c527-45d0-bedc-a2df8ac70f26-kube-api-access-968n8\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:32:51.813857 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.813821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9n2g\" (UniqueName: \"kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.814069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.813866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.814069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.813885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.814069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.813941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.814069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.814029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.814069 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.814064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915301 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9n2g\" (UniqueName: \"kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915301 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915807 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.915923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.915877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.917700 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.917679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.917962 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.917944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.924058 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.924037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9n2g\" (UniqueName: \"kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g\") pod \"owner-v1-kserve-75579ff5bc-fghjl\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:51.994453 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:51.994361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:32:52.024779 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.024746    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" containerID="a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45" exitCode=0
Sep 16 13:32:52.024939 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.024793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"
Sep 16 13:32:52.024939 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.024826    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" event={"ID":"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab","Type":"ContainerDied","Data":"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"}
Sep 16 13:32:52.024939 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.024880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm" event={"ID":"5a9b58dc-a5c6-411d-bc22-c4d35ad96aab","Type":"ContainerDied","Data":"8152cd64edf414cc34f7c74a9f9face4a4bd545096eb6a058cec0681a5eb61db"}
Sep 16 13:32:52.024939 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.024910    2579 scope.go:117] "RemoveContainer" containerID="a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"
Sep 16 13:32:52.026322 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.026301    2579 generic.go:356] "Generic (PLEG): container finished" podID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" containerID="87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be" exitCode=0
Sep 16 13:32:52.026435 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.026347    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"
Sep 16 13:32:52.026435 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.026365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" event={"ID":"60c7c1b6-c527-45d0-bedc-a2df8ac70f26","Type":"ContainerDied","Data":"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"}
Sep 16 13:32:52.026435 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.026403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj" event={"ID":"60c7c1b6-c527-45d0-bedc-a2df8ac70f26","Type":"ContainerDied","Data":"a057ece7bc188f5158011a202affae3a0bf3fef1f7a75fed5c3fe85a71042ac9"}
Sep 16 13:32:52.035504 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.035275    2579 scope.go:117] "RemoveContainer" containerID="a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"
Sep 16 13:32:52.035914 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:32:52.035870    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45\": container with ID starting with a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45 not found: ID does not exist" containerID="a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"
Sep 16 13:32:52.036002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.035906    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45"} err="failed to get container status \"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45\": rpc error: code = NotFound desc = could not find container \"a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45\": container with ID starting with a1308bac0ade88449744c59e89b4c4676885c85704c6a842f4a32331dd1a3d45 not found: ID does not exist"
Sep 16 13:32:52.036002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.035932    2579 scope.go:117] "RemoveContainer" containerID="87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"
Sep 16 13:32:52.045968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.045471    2579 scope.go:117] "RemoveContainer" containerID="87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"
Sep 16 13:32:52.046136 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:32:52.045965    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be\": container with ID starting with 87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be not found: ID does not exist" containerID="87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"
Sep 16 13:32:52.046136 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.045999    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be"} err="failed to get container status \"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be\": rpc error: code = NotFound desc = could not find container \"87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be\": container with ID starting with 87948088f676c4d5e126771dc20e83bb852c06eeba6d7197aef0903bfbd5b3be not found: ID does not exist"
Sep 16 13:32:52.051061 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.051034    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:32:52.061532 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.061503    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-gsmpm"]
Sep 16 13:32:52.074744 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.074694    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:32:52.079334 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.079263    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-vzxbj"]
Sep 16 13:32:52.274113 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.274088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:32:52.278082 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:32:52.278047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee3bac0_3fc1_46a0_9558_38ad90827cda.slice/crio-552caab3456de9818b5ec90b6aecfc5a2f5aceeacb84bb0ef0aa06925e19590f WatchSource:0}: Error finding container 552caab3456de9818b5ec90b6aecfc5a2f5aceeacb84bb0ef0aa06925e19590f: Status 404 returned error can't find the container with id 552caab3456de9818b5ec90b6aecfc5a2f5aceeacb84bb0ef0aa06925e19590f
Sep 16 13:32:52.461473 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.461432    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9b58dc-a5c6-411d-bc22-c4d35ad96aab" path="/var/lib/kubelet/pods/5a9b58dc-a5c6-411d-bc22-c4d35ad96aab/volumes"
Sep 16 13:32:52.462107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:52.462076    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c7c1b6-c527-45d0-bedc-a2df8ac70f26" path="/var/lib/kubelet/pods/60c7c1b6-c527-45d0-bedc-a2df8ac70f26/volumes"
Sep 16 13:32:53.034095 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:53.034058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" event={"ID":"3ee3bac0-3fc1-46a0-9558-38ad90827cda","Type":"ContainerStarted","Data":"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"}
Sep 16 13:32:53.034275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:53.034110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" event={"ID":"3ee3bac0-3fc1-46a0-9558-38ad90827cda","Type":"ContainerStarted","Data":"552caab3456de9818b5ec90b6aecfc5a2f5aceeacb84bb0ef0aa06925e19590f"}
Sep 16 13:32:53.055713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:53.055654    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" podStartSLOduration=2.055634195 podStartE2EDuration="2.055634195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:53.054612551 +0000 UTC m=+2679.093411675" watchObservedRunningTime="2026-09-16 13:32:53.055634195 +0000 UTC m=+2679.094433319"
Sep 16 13:32:53.995202 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:32:53.995170    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:33:01.994822 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:01.994782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:33:02.007634 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:02.007607    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:33:02.083581 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:02.083540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:33:11.586430 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.586350    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:33:11.589778 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.589760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.592336 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.592318    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 13:33:11.600091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.600067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:33:11.789176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.789379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.789379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.789379 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.789513 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6qjh\" (UniqueName: \"kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.789513 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.789423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890478 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890478 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6qjh\" (UniqueName: \"kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890478 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890478 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890711 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.890786 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.890772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.892671 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.892651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.892875 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.892856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.898036 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.898013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6qjh\" (UniqueName: \"kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh\") pod \"owner-v2-kserve-5cf8959f95-mlz8r\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:11.899897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:11.899876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:12.171188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:12.171157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:33:12.173406 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:33:12.173376    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19050b6_2952_47e5_977e_86542e1b218a.slice/crio-082ada5a67a77eefc3dbaac610d092634e3f91bab33b4016b8e211749aab7c03 WatchSource:0}: Error finding container 082ada5a67a77eefc3dbaac610d092634e3f91bab33b4016b8e211749aab7c03: Status 404 returned error can't find the container with id 082ada5a67a77eefc3dbaac610d092634e3f91bab33b4016b8e211749aab7c03
Sep 16 13:33:13.119541 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:13.119502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" event={"ID":"f19050b6-2952-47e5-977e-86542e1b218a","Type":"ContainerStarted","Data":"7b482ff6acb925e7dc209ea8910453dede8b0c081d12c524904054a558749fba"}
Sep 16 13:33:13.119974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:13.119594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" event={"ID":"f19050b6-2952-47e5-977e-86542e1b218a","Type":"ContainerStarted","Data":"082ada5a67a77eefc3dbaac610d092634e3f91bab33b4016b8e211749aab7c03"}
Sep 16 13:33:13.137895 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:13.137838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" podStartSLOduration=2.137822288 podStartE2EDuration="2.137822288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:13.135658787 +0000 UTC m=+2699.174457911" watchObservedRunningTime="2026-09-16 13:33:13.137822288 +0000 UTC m=+2699.176621411"
Sep 16 13:33:13.900514 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:13.900473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:13.990963 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:13.990933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:33:14.540070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:14.540037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:33:14.552252 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:14.552214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:33:21.900041 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:21.899997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:21.912400 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:21.912371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:33:22.168669 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:33:22.168572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:34:05.158079 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.158044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:34:05.161737 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.161711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.166641 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.166609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 13:34:05.166765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.166656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-crfkc\""
Sep 16 13:34:05.172460 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.172436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:34:05.335191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.335157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.335435 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.335262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj25w\" (UniqueName: \"kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.436191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.436099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj25w\" (UniqueName: \"kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.436191 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.436169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.438713 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.438686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.444679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.444655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj25w\" (UniqueName: \"kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.473718 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.473690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:05.746704 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:05.746676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:34:05.749219 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:34:05.749189    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e794d7_c1c1_4014_b60b_b2cc301d4141.slice/crio-9f27204982c0def0901fe5069a608266e4f3fb0a88f5da65fda3417294eeff9a WatchSource:0}: Error finding container 9f27204982c0def0901fe5069a608266e4f3fb0a88f5da65fda3417294eeff9a: Status 404 returned error can't find the container with id 9f27204982c0def0901fe5069a608266e4f3fb0a88f5da65fda3417294eeff9a
Sep 16 13:34:06.343747 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:06.343704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" event={"ID":"29e794d7-c1c1-4014-b60b-b2cc301d4141","Type":"ContainerStarted","Data":"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"}
Sep 16 13:34:06.344265 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:06.343771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" event={"ID":"29e794d7-c1c1-4014-b60b-b2cc301d4141","Type":"ContainerStarted","Data":"9f27204982c0def0901fe5069a608266e4f3fb0a88f5da65fda3417294eeff9a"}
Sep 16 13:34:06.344265 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:06.343800    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:06.363438 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:06.363376    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podStartSLOduration=1.363358614 podStartE2EDuration="1.363358614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:06.362129397 +0000 UTC m=+2752.400928520" watchObservedRunningTime="2026-09-16 13:34:06.363358614 +0000 UTC m=+2752.402157736"
Sep 16 13:34:22.711073 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:22.711033    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:34:37.350307 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:37.350224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:34:37.731505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:37.731460    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" podUID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" containerName="main" containerID="cri-o://68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d" gracePeriod=45
Sep 16 13:34:37.980116 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:37.980085    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:34:38.026063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.025974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026063 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9n2g\" (UniqueName: \"kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home\") pod \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\" (UID: \"3ee3bac0-3fc1-46a0-9558-38ad90827cda\") "
Sep 16 13:34:38.026399 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:38.026587 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:38.026655 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.026586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home" (OuterVolumeSpecName: "home") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:38.028465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.028435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm" (OuterVolumeSpecName: "dshm") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:38.028465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.028455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:34:38.028683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.028445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g" (OuterVolumeSpecName: "kube-api-access-g9n2g") pod "3ee3bac0-3fc1-46a0-9558-38ad90827cda" (UID: "3ee3bac0-3fc1-46a0-9558-38ad90827cda"). InnerVolumeSpecName "kube-api-access-g9n2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:34:38.127244 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127209    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.127244 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127237    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.127244 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127249    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.127495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127257    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.127495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127265    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee3bac0-3fc1-46a0-9558-38ad90827cda-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.127495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.127274    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9n2g\" (UniqueName: \"kubernetes.io/projected/3ee3bac0-3fc1-46a0-9558-38ad90827cda-kube-api-access-g9n2g\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:34:38.483127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.483087    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" containerID="68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d" exitCode=0
Sep 16 13:34:38.483501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.483127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" event={"ID":"3ee3bac0-3fc1-46a0-9558-38ad90827cda","Type":"ContainerDied","Data":"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"}
Sep 16 13:34:38.483501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.483147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"
Sep 16 13:34:38.483501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.483214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl" event={"ID":"3ee3bac0-3fc1-46a0-9558-38ad90827cda","Type":"ContainerDied","Data":"552caab3456de9818b5ec90b6aecfc5a2f5aceeacb84bb0ef0aa06925e19590f"}
Sep 16 13:34:38.483501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.483230    2579 scope.go:117] "RemoveContainer" containerID="68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"
Sep 16 13:34:38.491536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.491520    2579 scope.go:117] "RemoveContainer" containerID="68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"
Sep 16 13:34:38.491835 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:34:38.491815    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d\": container with ID starting with 68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d not found: ID does not exist" containerID="68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"
Sep 16 13:34:38.491885 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.491841    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d"} err="failed to get container status \"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d\": rpc error: code = NotFound desc = could not find container \"68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d\": container with ID starting with 68bb495800739265a08931cec39893d561d66bf0b9c9ab7c6e6e0741066fb44d not found: ID does not exist"
Sep 16 13:34:38.501551 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.501528    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:34:38.504997 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:38.504972    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-fghjl"]
Sep 16 13:34:40.202962 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:40.202928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:34:40.458266 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:40.458183    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" path="/var/lib/kubelet/pods/3ee3bac0-3fc1-46a0-9558-38ad90827cda/volumes"
Sep 16 13:34:47.349791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:47.349753    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:34:51.374681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:51.374639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:34:55.222645 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:55.222575    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main" containerID="cri-o://3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643" gracePeriod=45
Sep 16 13:34:57.350802 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:34:57.350763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:06.401257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.401187    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" podUID="f19050b6-2952-47e5-977e-86542e1b218a" containerName="main" containerID="cri-o://7b482ff6acb925e7dc209ea8910453dede8b0c081d12c524904054a558749fba" gracePeriod=45
Sep 16 13:35:06.612538 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.612504    2579 generic.go:356] "Generic (PLEG): container finished" podID="f19050b6-2952-47e5-977e-86542e1b218a" containerID="7b482ff6acb925e7dc209ea8910453dede8b0c081d12c524904054a558749fba" exitCode=0
Sep 16 13:35:06.612751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.612589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" event={"ID":"f19050b6-2952-47e5-977e-86542e1b218a","Type":"ContainerDied","Data":"7b482ff6acb925e7dc209ea8910453dede8b0c081d12c524904054a558749fba"}
Sep 16 13:35:06.648065 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.648041    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:35:06.669598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.669598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.669598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.670019 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6qjh\" (UniqueName: \"kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.670019 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.670019 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.669730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache\") pod \"f19050b6-2952-47e5-977e-86542e1b218a\" (UID: \"f19050b6-2952-47e5-977e-86542e1b218a\") "
Sep 16 13:35:06.670178 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.670140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:06.670178 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.670154    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home" (OuterVolumeSpecName: "home") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:06.670292 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.670226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:06.671825 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.671795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm" (OuterVolumeSpecName: "dshm") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:06.672023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.671999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:35:06.672267 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.672244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh" (OuterVolumeSpecName: "kube-api-access-t6qjh") pod "f19050b6-2952-47e5-977e-86542e1b218a" (UID: "f19050b6-2952-47e5-977e-86542e1b218a"). InnerVolumeSpecName "kube-api-access-t6qjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:35:06.770870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770833    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:06.770870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770865    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:06.770870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770874    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f19050b6-2952-47e5-977e-86542e1b218a-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:06.770870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770881    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:06.771130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770890    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f19050b6-2952-47e5-977e-86542e1b218a-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:06.771130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:06.770901    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6qjh\" (UniqueName: \"kubernetes.io/projected/f19050b6-2952-47e5-977e-86542e1b218a-kube-api-access-t6qjh\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:07.350625 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.350555    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:07.350830 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.350703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:35:07.618140 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.618059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"
Sep 16 13:35:07.618140 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.618086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r" event={"ID":"f19050b6-2952-47e5-977e-86542e1b218a","Type":"ContainerDied","Data":"082ada5a67a77eefc3dbaac610d092634e3f91bab33b4016b8e211749aab7c03"}
Sep 16 13:35:07.618553 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.618147    2579 scope.go:117] "RemoveContainer" containerID="7b482ff6acb925e7dc209ea8910453dede8b0c081d12c524904054a558749fba"
Sep 16 13:35:07.639912 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.639882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:35:07.644057 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:07.644036    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mlz8r"]
Sep 16 13:35:08.457515 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:08.457479    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19050b6-2952-47e5-977e-86542e1b218a" path="/var/lib/kubelet/pods/f19050b6-2952-47e5-977e-86542e1b218a/volumes"
Sep 16 13:35:17.349822 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:17.349774    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:25.467700 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.467662    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:35:25.524193 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.524094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj25w\" (UniqueName: \"kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w\") pod \"29e794d7-c1c1-4014-b60b-b2cc301d4141\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") "
Sep 16 13:35:25.524193 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.524168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs\") pod \"29e794d7-c1c1-4014-b60b-b2cc301d4141\" (UID: \"29e794d7-c1c1-4014-b60b-b2cc301d4141\") "
Sep 16 13:35:25.526496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.526452    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29e794d7-c1c1-4014-b60b-b2cc301d4141" (UID: "29e794d7-c1c1-4014-b60b-b2cc301d4141"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:35:25.526660 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.526491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w" (OuterVolumeSpecName: "kube-api-access-mj25w") pod "29e794d7-c1c1-4014-b60b-b2cc301d4141" (UID: "29e794d7-c1c1-4014-b60b-b2cc301d4141"). InnerVolumeSpecName "kube-api-access-mj25w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:35:25.625022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.624974    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e794d7-c1c1-4014-b60b-b2cc301d4141-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:25.625022 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.625016    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj25w\" (UniqueName: \"kubernetes.io/projected/29e794d7-c1c1-4014-b60b-b2cc301d4141-kube-api-access-mj25w\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:35:25.691960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.691920    2579 generic.go:356] "Generic (PLEG): container finished" podID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerID="3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643" exitCode=0
Sep 16 13:35:25.692168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.691972    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"
Sep 16 13:35:25.692168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.692017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" event={"ID":"29e794d7-c1c1-4014-b60b-b2cc301d4141","Type":"ContainerDied","Data":"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"}
Sep 16 13:35:25.692168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.692094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4" event={"ID":"29e794d7-c1c1-4014-b60b-b2cc301d4141","Type":"ContainerDied","Data":"9f27204982c0def0901fe5069a608266e4f3fb0a88f5da65fda3417294eeff9a"}
Sep 16 13:35:25.692168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.692116    2579 scope.go:117] "RemoveContainer" containerID="3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"
Sep 16 13:35:25.700991 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.700971    2579 scope.go:117] "RemoveContainer" containerID="3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"
Sep 16 13:35:25.701305 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:35:25.701288    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643\": container with ID starting with 3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643 not found: ID does not exist" containerID="3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"
Sep 16 13:35:25.701353 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.701315    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643"} err="failed to get container status \"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643\": rpc error: code = NotFound desc = could not find container \"3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643\": container with ID starting with 3e27eaec05cf21cd7885796865cdc2ced7431280d3f822e4d79c7c83a0fb8643 not found: ID does not exist"
Sep 16 13:35:25.715062 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.715032    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:35:25.718919 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:25.718894    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche824h4"]
Sep 16 13:35:26.458281 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:26.458243    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" path="/var/lib/kubelet/pods/29e794d7-c1c1-4014-b60b-b2cc301d4141/volumes"
Sep 16 13:35:30.642015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:30.641977    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:35:33.990587 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:33.990525    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:43.990325 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:43.990276    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:45.661923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:45.661882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" containerID="cri-o://ed49bc83108918f8dbb0bd5341e2d6e9107c9c1354ce7fb4310909f9c719b371" gracePeriod=45
Sep 16 13:35:53.990807 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:53.990764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:35:53.991228 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:35:53.990871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:36:03.991197 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:03.991156    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:36:13.990845 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:13.990799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:36:15.903964 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.903932    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a875713-0e4d-4962-b8e0-b13181a81282" containerID="ed49bc83108918f8dbb0bd5341e2d6e9107c9c1354ce7fb4310909f9c719b371" exitCode=0
Sep 16 13:36:15.904356 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.903978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" event={"ID":"6a875713-0e4d-4962-b8e0-b13181a81282","Type":"ContainerDied","Data":"ed49bc83108918f8dbb0bd5341e2d6e9107c9c1354ce7fb4310909f9c719b371"}
Sep 16 13:36:15.904356 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.904047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn" event={"ID":"6a875713-0e4d-4962-b8e0-b13181a81282","Type":"ContainerDied","Data":"dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2"}
Sep 16 13:36:15.904356 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.904072    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbcabc75ad939a029b335848a8aa1eaecb509b231e093d858e8ae2df8f140bf2"
Sep 16 13:36:15.907364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.907345    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:36:15.946155 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.946044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs\") pod \"6a875713-0e4d-4962-b8e0-b13181a81282\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") "
Sep 16 13:36:15.946155 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.946121    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2xx6\" (UniqueName: \"kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6\") pod \"6a875713-0e4d-4962-b8e0-b13181a81282\" (UID: \"6a875713-0e4d-4962-b8e0-b13181a81282\") "
Sep 16 13:36:15.948296 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.948268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a875713-0e4d-4962-b8e0-b13181a81282" (UID: "6a875713-0e4d-4962-b8e0-b13181a81282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:15.948296 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:15.948279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6" (OuterVolumeSpecName: "kube-api-access-s2xx6") pod "6a875713-0e4d-4962-b8e0-b13181a81282" (UID: "6a875713-0e4d-4962-b8e0-b13181a81282"). InnerVolumeSpecName "kube-api-access-s2xx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:16.047449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:16.047408    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xx6\" (UniqueName: \"kubernetes.io/projected/6a875713-0e4d-4962-b8e0-b13181a81282-kube-api-access-s2xx6\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:36:16.047449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:16.047440    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a875713-0e4d-4962-b8e0-b13181a81282-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:36:16.908091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:16.908057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"
Sep 16 13:36:16.925364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:16.925331    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:36:16.929734 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:16.929705    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehx9jn"]
Sep 16 13:36:18.458234 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:18.458197    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" path="/var/lib/kubelet/pods/6a875713-0e4d-4962-b8e0-b13181a81282/volumes"
Sep 16 13:36:47.921115 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921349    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921360    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921369    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19050b6-2952-47e5-977e-86542e1b218a" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921375    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19050b6-2952-47e5-977e-86542e1b218a" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921390    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921397    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921411    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921416    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921460    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e794d7-c1c1-4014-b60b-b2cc301d4141" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921466    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ee3bac0-3fc1-46a0-9558-38ad90827cda" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921471    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a875713-0e4d-4962-b8e0-b13181a81282" containerName="main"
Sep 16 13:36:47.921656 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.921477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19050b6-2952-47e5-977e-86542e1b218a" containerName="main"
Sep 16 13:36:47.924548 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.924530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:47.928644 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.928610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 13:36:47.928765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.928675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ncqxw\""
Sep 16 13:36:47.928765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.928686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 13:36:47.928765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.928704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 13:36:47.934857 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:47.934832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:36:48.012134 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.012339 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.012339 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.012339 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.012339 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5cj\" (UniqueName: \"kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.012492 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.012332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5cj\" (UniqueName: \"kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.113870 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.113815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.114086 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.114057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.114086 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.114076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.114164 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.114127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.116181 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.116158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.116336 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.116318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.124259 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.124234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5cj\" (UniqueName: \"kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.168397 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.168363    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:36:48.171842 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.171776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.174770 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.174745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-fhzgt\""
Sep 16 13:36:48.185150 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.185115    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:36:48.235265 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.235216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:48.316316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.316280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2n67\" (UniqueName: \"kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.316518 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.316327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.417731 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.417689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n67\" (UniqueName: \"kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.417731 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.417734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.420352 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.420323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.428019 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.427934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2n67\" (UniqueName: \"kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.482760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.482719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:48.515248 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.515220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:36:48.517452 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:36:48.517413    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772e1384_de72_4af9_a24d_1b1afeee2c13.slice/crio-f675ef34050431d0ff67d26273f580bb59c300605624a94d24ec2ccf40662d32 WatchSource:0}: Error finding container f675ef34050431d0ff67d26273f580bb59c300605624a94d24ec2ccf40662d32: Status 404 returned error can't find the container with id f675ef34050431d0ff67d26273f580bb59c300605624a94d24ec2ccf40662d32
Sep 16 13:36:48.764598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:48.764557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:36:48.766954 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:36:48.766922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb590f032_490b_47d6_af32_e5b1aeec9431.slice/crio-cd75d6b710e531f2f4e9b391c0c3f33ed5bba3eb10bf870ed42924a9824949eb WatchSource:0}: Error finding container cd75d6b710e531f2f4e9b391c0c3f33ed5bba3eb10bf870ed42924a9824949eb: Status 404 returned error can't find the container with id cd75d6b710e531f2f4e9b391c0c3f33ed5bba3eb10bf870ed42924a9824949eb
Sep 16 13:36:49.049128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.049026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" event={"ID":"772e1384-de72-4af9-a24d-1b1afeee2c13","Type":"ContainerStarted","Data":"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"}
Sep 16 13:36:49.049128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.049102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" event={"ID":"772e1384-de72-4af9-a24d-1b1afeee2c13","Type":"ContainerStarted","Data":"f675ef34050431d0ff67d26273f580bb59c300605624a94d24ec2ccf40662d32"}
Sep 16 13:36:49.050962 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.050930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" event={"ID":"b590f032-490b-47d6-af32-e5b1aeec9431","Type":"ContainerStarted","Data":"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"}
Sep 16 13:36:49.051120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.050978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" event={"ID":"b590f032-490b-47d6-af32-e5b1aeec9431","Type":"ContainerStarted","Data":"cd75d6b710e531f2f4e9b391c0c3f33ed5bba3eb10bf870ed42924a9824949eb"}
Sep 16 13:36:49.051120 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.051081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:36:49.083765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.083697    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" podStartSLOduration=2.083674207 podStartE2EDuration="2.083674207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:49.082526916 +0000 UTC m=+2915.121326043" watchObservedRunningTime="2026-09-16 13:36:49.083674207 +0000 UTC m=+2915.122473333"
Sep 16 13:36:49.114205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.114144    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podStartSLOduration=1.114121457 podStartE2EDuration="1.114121457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:49.113277506 +0000 UTC m=+2915.152076630" watchObservedRunningTime="2026-09-16 13:36:49.114121457 +0000 UTC m=+2915.152920582"
Sep 16 13:36:49.235524 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:49.235475    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:58.236331 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:58.236286    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:58.249231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:58.249197    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:36:59.114239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:36:59.114199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:37:20.059607 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:20.059553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:37:31.065216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.065184    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:37:31.068484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.068457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.072343 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.072311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-qg7n8\""
Sep 16 13:37:31.072343 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.072326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 13:37:31.081937 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.081911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:37:31.173841 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.173800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.174054 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.173939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hpm\" (UniqueName: \"kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.275222 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.275181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hpm\" (UniqueName: \"kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.275449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.275237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.277760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.277739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.284839 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.284814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hpm\" (UniqueName: \"kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.378816 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.378705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:31.649738 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:31.649708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:37:31.652327 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:37:31.652297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873d7a61_6b92_4da2_867f_568735086c73.slice/crio-6d54544c1e24fd78d4726ac35a3abf1bc004f379f13ceba146707ef4a80e4fbe WatchSource:0}: Error finding container 6d54544c1e24fd78d4726ac35a3abf1bc004f379f13ceba146707ef4a80e4fbe: Status 404 returned error can't find the container with id 6d54544c1e24fd78d4726ac35a3abf1bc004f379f13ceba146707ef4a80e4fbe
Sep 16 13:37:32.251814 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:32.251753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" event={"ID":"873d7a61-6b92-4da2-867f-568735086c73","Type":"ContainerStarted","Data":"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"}
Sep 16 13:37:32.252316 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:32.251901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" event={"ID":"873d7a61-6b92-4da2-867f-568735086c73","Type":"ContainerStarted","Data":"6d54544c1e24fd78d4726ac35a3abf1bc004f379f13ceba146707ef4a80e4fbe"}
Sep 16 13:37:32.252661 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:32.252619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:37:32.276723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:37:32.276660    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podStartSLOduration=1.276642796 podStartE2EDuration="1.276642796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:32.275093922 +0000 UTC m=+2958.313893046" watchObservedRunningTime="2026-09-16 13:37:32.276642796 +0000 UTC m=+2958.315441919"
Sep 16 13:38:04.262306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:04.262228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:38:05.877623 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:05.877586    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:38:14.262479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:14.262434    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:38:14.570606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:14.570505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:38:14.578059 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:14.578022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:38:20.898446 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:20.898376    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main" containerID="cri-o://c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d" gracePeriod=45
Sep 16 13:38:24.262097 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:24.262059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:38:34.262849 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:34.262804    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:38:34.263388 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:34.262933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:38:44.262164 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:44.262123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:38:51.153569 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.153524    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:38:51.244427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.244334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8hpm\" (UniqueName: \"kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm\") pod \"873d7a61-6b92-4da2-867f-568735086c73\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") "
Sep 16 13:38:51.244427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.244417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs\") pod \"873d7a61-6b92-4da2-867f-568735086c73\" (UID: \"873d7a61-6b92-4da2-867f-568735086c73\") "
Sep 16 13:38:51.246683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.246651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "873d7a61-6b92-4da2-867f-568735086c73" (UID: "873d7a61-6b92-4da2-867f-568735086c73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:38:51.246795 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.246699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm" (OuterVolumeSpecName: "kube-api-access-g8hpm") pod "873d7a61-6b92-4da2-867f-568735086c73" (UID: "873d7a61-6b92-4da2-867f-568735086c73"). InnerVolumeSpecName "kube-api-access-g8hpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:38:51.344943 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.344904    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hpm\" (UniqueName: \"kubernetes.io/projected/873d7a61-6b92-4da2-867f-568735086c73-kube-api-access-g8hpm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:51.344943 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.344937    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873d7a61-6b92-4da2-867f-568735086c73-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:51.589637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.589525    2579 generic.go:356] "Generic (PLEG): container finished" podID="873d7a61-6b92-4da2-867f-568735086c73" containerID="c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d" exitCode=0
Sep 16 13:38:51.589637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.589613    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"
Sep 16 13:38:51.589637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.589617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" event={"ID":"873d7a61-6b92-4da2-867f-568735086c73","Type":"ContainerDied","Data":"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"}
Sep 16 13:38:51.589942 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.589683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t" event={"ID":"873d7a61-6b92-4da2-867f-568735086c73","Type":"ContainerDied","Data":"6d54544c1e24fd78d4726ac35a3abf1bc004f379f13ceba146707ef4a80e4fbe"}
Sep 16 13:38:51.589942 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.589714    2579 scope.go:117] "RemoveContainer" containerID="c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"
Sep 16 13:38:51.599078 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.599055    2579 scope.go:117] "RemoveContainer" containerID="c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"
Sep 16 13:38:51.599376 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:38:51.599355    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d\": container with ID starting with c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d not found: ID does not exist" containerID="c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"
Sep 16 13:38:51.599440 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.599390    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d"} err="failed to get container status \"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d\": rpc error: code = NotFound desc = could not find container \"c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d\": container with ID starting with c1dba66f2d4e49672206dd2bf760166bd4d5aec40e5d26fd5cde52ea79522c6d not found: ID does not exist"
Sep 16 13:38:51.614591 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.614531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:38:51.620917 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:51.620882    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebfg7t"]
Sep 16 13:38:52.458127 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:52.458082    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873d7a61-6b92-4da2-867f-568735086c73" path="/var/lib/kubelet/pods/873d7a61-6b92-4da2-867f-568735086c73/volumes"
Sep 16 13:38:56.128298 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.128262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:38:56.130359 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.130324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:38:56.140238 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:38:56.140192    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:38:56.140238 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:56.140238 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:56.140238 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe" cmd=["/bin/sleep","15"]
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.140259    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" message="" err=<
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:38:56.140277    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerName="main" containerID="cri-o://ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"
Sep 16 13:38:56.140579 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.140338    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerName="main" containerID="cri-o://ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe" gracePeriod=60
Sep 16 13:38:56.491130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.491102    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:38:56.588275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588451 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588451 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588451 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588351    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home" (OuterVolumeSpecName: "home") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:56.588610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:56.588785 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw5cj\" (UniqueName: \"kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj\") pod \"772e1384-de72-4af9-a24d-1b1afeee2c13\" (UID: \"772e1384-de72-4af9-a24d-1b1afeee2c13\") "
Sep 16 13:38:56.588839 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588797    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache" (OuterVolumeSpecName: "model-cache") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:56.588885 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588857    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:56.588885 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.588876    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:56.590572 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.590538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm" (OuterVolumeSpecName: "dshm") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:56.590673 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.590650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj" (OuterVolumeSpecName: "kube-api-access-cw5cj") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "kube-api-access-cw5cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:38:56.590716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.590668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "772e1384-de72-4af9-a24d-1b1afeee2c13" (UID: "772e1384-de72-4af9-a24d-1b1afeee2c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:38:56.611633 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.611602    2579 generic.go:356] "Generic (PLEG): container finished" podID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerID="ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe" exitCode=0
Sep 16 13:38:56.611766 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.611651    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"
Sep 16 13:38:56.611766 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.611683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" event={"ID":"772e1384-de72-4af9-a24d-1b1afeee2c13","Type":"ContainerDied","Data":"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"}
Sep 16 13:38:56.611766 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.611743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7" event={"ID":"772e1384-de72-4af9-a24d-1b1afeee2c13","Type":"ContainerDied","Data":"f675ef34050431d0ff67d26273f580bb59c300605624a94d24ec2ccf40662d32"}
Sep 16 13:38:56.611880 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.611774    2579 scope.go:117] "RemoveContainer" containerID="ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"
Sep 16 13:38:56.620777 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.620755    2579 scope.go:117] "RemoveContainer" containerID="ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"
Sep 16 13:38:56.621083 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:38:56.621062    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe\": container with ID starting with ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe not found: ID does not exist" containerID="ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"
Sep 16 13:38:56.621150 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.621091    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe"} err="failed to get container status \"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe\": rpc error: code = NotFound desc = could not find container \"ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe\": container with ID starting with ad9d1944ad5e201d2f00f6e201374e42775138f7d76a63e79fbdb760e59079fe not found: ID does not exist"
Sep 16 13:38:56.634182 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.634154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:38:56.642516 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.642488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796925cr7"]
Sep 16 13:38:56.690003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.689927    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:56.690003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.689956    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5cj\" (UniqueName: \"kubernetes.io/projected/772e1384-de72-4af9-a24d-1b1afeee2c13-kube-api-access-cw5cj\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:56.690003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.689967    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772e1384-de72-4af9-a24d-1b1afeee2c13-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:56.690003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:56.689975    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772e1384-de72-4af9-a24d-1b1afeee2c13-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:38:58.457483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:38:58.457448    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" path="/var/lib/kubelet/pods/772e1384-de72-4af9-a24d-1b1afeee2c13/volumes"
Sep 16 13:39:00.060091 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:00.060047    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:39:10.060068 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:10.059975    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:39:11.148510 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:11.148470    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" containerID="cri-o://3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f" gracePeriod=45
Sep 16 13:39:14.772990 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:14.772958    2579 scope.go:117] "RemoveContainer" containerID="ed49bc83108918f8dbb0bd5341e2d6e9107c9c1354ce7fb4310909f9c719b371"
Sep 16 13:39:20.059668 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:20.059631    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:39:20.060052 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:20.059740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:39:30.058994 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:30.058955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:39:40.059927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:40.059881    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:39:41.398384 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.398356    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:39:41.455351 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.455257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2n67\" (UniqueName: \"kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67\") pod \"b590f032-490b-47d6-af32-e5b1aeec9431\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") "
Sep 16 13:39:41.455351 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.455333    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs\") pod \"b590f032-490b-47d6-af32-e5b1aeec9431\" (UID: \"b590f032-490b-47d6-af32-e5b1aeec9431\") "
Sep 16 13:39:41.457420 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.457385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67" (OuterVolumeSpecName: "kube-api-access-n2n67") pod "b590f032-490b-47d6-af32-e5b1aeec9431" (UID: "b590f032-490b-47d6-af32-e5b1aeec9431"). InnerVolumeSpecName "kube-api-access-n2n67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:41.457545 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.457466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b590f032-490b-47d6-af32-e5b1aeec9431" (UID: "b590f032-490b-47d6-af32-e5b1aeec9431"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:39:41.556515 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.556478    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2n67\" (UniqueName: \"kubernetes.io/projected/b590f032-490b-47d6-af32-e5b1aeec9431-kube-api-access-n2n67\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.556515 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.556512    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b590f032-490b-47d6-af32-e5b1aeec9431-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.797479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.797381    2579 generic.go:356] "Generic (PLEG): container finished" podID="b590f032-490b-47d6-af32-e5b1aeec9431" containerID="3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f" exitCode=0
Sep 16 13:39:41.797479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.797423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"
Sep 16 13:39:41.797716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.797479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" event={"ID":"b590f032-490b-47d6-af32-e5b1aeec9431","Type":"ContainerDied","Data":"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"}
Sep 16 13:39:41.797716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.797540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n" event={"ID":"b590f032-490b-47d6-af32-e5b1aeec9431","Type":"ContainerDied","Data":"cd75d6b710e531f2f4e9b391c0c3f33ed5bba3eb10bf870ed42924a9824949eb"}
Sep 16 13:39:41.797716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.797585    2579 scope.go:117] "RemoveContainer" containerID="3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"
Sep 16 13:39:41.817445 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.817423    2579 scope.go:117] "RemoveContainer" containerID="3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"
Sep 16 13:39:41.817850 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:39:41.817827    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f\": container with ID starting with 3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f not found: ID does not exist" containerID="3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"
Sep 16 13:39:41.817940 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.817862    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f"} err="failed to get container status \"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f\": rpc error: code = NotFound desc = could not find container \"3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f\": container with ID starting with 3958121e631570297554eb9986c9f692a574a1bfeabd3d24319e6baccbd5164f not found: ID does not exist"
Sep 16 13:39:41.827725 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.827695    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:39:41.829828 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:41.829801    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecfc7n"]
Sep 16 13:39:42.458434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:42.458396    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" path="/var/lib/kubelet/pods/b590f032-490b-47d6-af32-e5b1aeec9431/volumes"
Sep 16 13:39:45.104925 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.104893    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105166    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105191    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105198    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105204    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105255    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="873d7a61-6b92-4da2-867f-568735086c73" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105264    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="772e1384-de72-4af9-a24d-1b1afeee2c13" containerName="main"
Sep 16 13:39:45.105324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.105271    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b590f032-490b-47d6-af32-e5b1aeec9431" containerName="main"
Sep 16 13:39:45.109324 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.109304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.113555 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.113529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-tld92\""
Sep 16 13:39:45.113701 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.113529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 13:39:45.113701 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.113648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 13:39:45.113927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.113909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 13:39:45.114002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.113927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-2p8hv\""
Sep 16 13:39:45.122108 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.122077    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:39:45.188038 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.187992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.188205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.188116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwjs\" (UniqueName: \"kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.288867 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.288825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwjs\" (UniqueName: \"kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.289048 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.288908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.291716 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.291685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.299124 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.299094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwjs\" (UniqueName: \"kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.419813 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.419779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.690771 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.690690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:39:45.693624 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:39:45.693592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c69a5e7_8b22_42a5_8d9f_0abfec53b47d.slice/crio-481635d0024191b3b4e3a4e8f1ade8d3ed34448e054296f40e006b5e76ad161b WatchSource:0}: Error finding container 481635d0024191b3b4e3a4e8f1ade8d3ed34448e054296f40e006b5e76ad161b: Status 404 returned error can't find the container with id 481635d0024191b3b4e3a4e8f1ade8d3ed34448e054296f40e006b5e76ad161b
Sep 16 13:39:45.816827 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.816786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" event={"ID":"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d","Type":"ContainerStarted","Data":"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"}
Sep 16 13:39:45.817070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.816859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" event={"ID":"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d","Type":"ContainerStarted","Data":"481635d0024191b3b4e3a4e8f1ade8d3ed34448e054296f40e006b5e76ad161b"}
Sep 16 13:39:45.817070 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.816968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:39:45.835819 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:39:45.835747    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podStartSLOduration=0.835726014 podStartE2EDuration="835.726014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:45.835139187 +0000 UTC m=+3091.873938312" watchObservedRunningTime="2026-09-16 13:39:45.835726014 +0000 UTC m=+3091.874525141"
Sep 16 13:40:09.545388 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.545303    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:40:09.548730 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.548710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.558995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.558970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 13:40:09.560364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.560333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 13:40:09.560364 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.560344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 13:40:09.561306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.561277    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-8jmxx\""
Sep 16 13:40:09.563508 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.563486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-zsl4k\""
Sep 16 13:40:09.566165 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.566141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:40:09.694804 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.694764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.694999 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.694853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf66m\" (UniqueName: \"kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.795820 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.795717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf66m\" (UniqueName: \"kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.795820 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.795800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.798337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.798309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.804860 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.804837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf66m\" (UniqueName: \"kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:09.859782 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:09.859745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:10.160592 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:10.160540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:40:10.164630 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:40:10.164592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dfcd55_0865_4af9_80e9_f9b725e5fe92.slice/crio-bf53ac1087d8dd98ee647889590959bd89fc72668f9a6a4f361179f6aae16840 WatchSource:0}: Error finding container bf53ac1087d8dd98ee647889590959bd89fc72668f9a6a4f361179f6aae16840: Status 404 returned error can't find the container with id bf53ac1087d8dd98ee647889590959bd89fc72668f9a6a4f361179f6aae16840
Sep 16 13:40:10.934783 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:10.934741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" event={"ID":"d9dfcd55-0865-4af9-80e9-f9b725e5fe92","Type":"ContainerStarted","Data":"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"}
Sep 16 13:40:10.935210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:10.934803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" event={"ID":"d9dfcd55-0865-4af9-80e9-f9b725e5fe92","Type":"ContainerStarted","Data":"bf53ac1087d8dd98ee647889590959bd89fc72668f9a6a4f361179f6aae16840"}
Sep 16 13:40:10.935210 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:10.934853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:40:10.954479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:10.954429    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podStartSLOduration=1.954414677 podStartE2EDuration="1.954414677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:10.953771885 +0000 UTC m=+3116.992571011" watchObservedRunningTime="2026-09-16 13:40:10.954414677 +0000 UTC m=+3116.993213800"
Sep 16 13:40:16.828696 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:16.828659    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:40:41.941141 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:40:41.941061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:41:52.867434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:41:52.867389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:41:56.829000 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:41:56.828954    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:04.161744 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:04.161702    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:42:06.828242 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:06.828150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:07.886709 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:07.886645    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" containerID="cri-o://8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08" gracePeriod=45
Sep 16 13:42:11.941947 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:11.941904    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:16.828306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:16.828262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:16.828776 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:16.828377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:42:19.183574 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:19.183516    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main" containerID="cri-o://b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7" gracePeriod=45
Sep 16 13:42:21.941501 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:21.941468    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:26.829137 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:26.829093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:31.941811 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:31.941764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:31.942224 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:31.941882    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:42:36.828801 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:36.828758    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:38.139141 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.139111    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:42:38.199173 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.199134    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slwjs\" (UniqueName: \"kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs\") pod \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") "
Sep 16 13:42:38.199345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.199197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs\") pod \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\" (UID: \"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d\") "
Sep 16 13:42:38.201498 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.201471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" (UID: "8c69a5e7-8b22-42a5-8d9f-0abfec53b47d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:42:38.201629 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.201467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs" (OuterVolumeSpecName: "kube-api-access-slwjs") pod "8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" (UID: "8c69a5e7-8b22-42a5-8d9f-0abfec53b47d"). InnerVolumeSpecName "kube-api-access-slwjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:42:38.300643 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.300603    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwjs\" (UniqueName: \"kubernetes.io/projected/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-kube-api-access-slwjs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:42:38.300643 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.300639    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:42:38.552951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.552914    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerID="8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08" exitCode=0
Sep 16 13:42:38.553128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.552960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"
Sep 16 13:42:38.553128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.552967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" event={"ID":"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d","Type":"ContainerDied","Data":"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"}
Sep 16 13:42:38.553128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.553021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7" event={"ID":"8c69a5e7-8b22-42a5-8d9f-0abfec53b47d","Type":"ContainerDied","Data":"481635d0024191b3b4e3a4e8f1ade8d3ed34448e054296f40e006b5e76ad161b"}
Sep 16 13:42:38.553128 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.553050    2579 scope.go:117] "RemoveContainer" containerID="8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"
Sep 16 13:42:38.561636 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.561614    2579 scope.go:117] "RemoveContainer" containerID="8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"
Sep 16 13:42:38.561918 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:42:38.561901    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08\": container with ID starting with 8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08 not found: ID does not exist" containerID="8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"
Sep 16 13:42:38.561961 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.561929    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08"} err="failed to get container status \"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08\": rpc error: code = NotFound desc = could not find container \"8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08\": container with ID starting with 8de45420a616b0ea3e1d2df805e2a7dafcb01359a51486c659d2a7f38c314d08 not found: ID does not exist"
Sep 16 13:42:38.569979 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.569954    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:42:38.573644 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:38.573625    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherghw7"]
Sep 16 13:42:40.458229 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:40.458192    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" path="/var/lib/kubelet/pods/8c69a5e7-8b22-42a5-8d9f-0abfec53b47d/volumes"
Sep 16 13:42:41.941982 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:41.941941    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:42:49.428793 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.428767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:42:49.486812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.486718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs\") pod \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") "
Sep 16 13:42:49.486812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.486788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf66m\" (UniqueName: \"kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m\") pod \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\" (UID: \"d9dfcd55-0865-4af9-80e9-f9b725e5fe92\") "
Sep 16 13:42:49.488966 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.488936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9dfcd55-0865-4af9-80e9-f9b725e5fe92" (UID: "d9dfcd55-0865-4af9-80e9-f9b725e5fe92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:42:49.489109 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.488984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m" (OuterVolumeSpecName: "kube-api-access-wf66m") pod "d9dfcd55-0865-4af9-80e9-f9b725e5fe92" (UID: "d9dfcd55-0865-4af9-80e9-f9b725e5fe92"). InnerVolumeSpecName "kube-api-access-wf66m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:42:49.587579 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.587526    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:42:49.587579 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.587556    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf66m\" (UniqueName: \"kubernetes.io/projected/d9dfcd55-0865-4af9-80e9-f9b725e5fe92-kube-api-access-wf66m\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:42:49.601232 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.601198    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerID="b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7" exitCode=0
Sep 16 13:42:49.601378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.601249    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"
Sep 16 13:42:49.601378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.601285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" event={"ID":"d9dfcd55-0865-4af9-80e9-f9b725e5fe92","Type":"ContainerDied","Data":"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"}
Sep 16 13:42:49.601378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.601349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777" event={"ID":"d9dfcd55-0865-4af9-80e9-f9b725e5fe92","Type":"ContainerDied","Data":"bf53ac1087d8dd98ee647889590959bd89fc72668f9a6a4f361179f6aae16840"}
Sep 16 13:42:49.601536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.601380    2579 scope.go:117] "RemoveContainer" containerID="b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"
Sep 16 13:42:49.610403 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.610384    2579 scope.go:117] "RemoveContainer" containerID="b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"
Sep 16 13:42:49.610680 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:42:49.610657    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7\": container with ID starting with b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7 not found: ID does not exist" containerID="b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"
Sep 16 13:42:49.610758 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.610687    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7"} err="failed to get container status \"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7\": rpc error: code = NotFound desc = could not find container \"b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7\": container with ID starting with b3d517af784bae4a7ff239761e53320ba117aec05212f0d46267d7c078f1cea7 not found: ID does not exist"
Sep 16 13:42:49.623241 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.623217    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:42:49.628954 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:49.628932    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqn777"]
Sep 16 13:42:50.458648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:42:50.458619    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" path="/var/lib/kubelet/pods/d9dfcd55-0865-4af9-80e9-f9b725e5fe92/volumes"
Sep 16 13:43:11.729695 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.729660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.729956    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main"
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.729971    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main"
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.729991    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main"
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.729997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main"
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.730043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c69a5e7-8b22-42a5-8d9f-0abfec53b47d" containerName="main"
Sep 16 13:43:11.730161 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.730053    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9dfcd55-0865-4af9-80e9-f9b725e5fe92" containerName="main"
Sep 16 13:43:11.732911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.732891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.735504 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.735478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-cgt4v\""
Sep 16 13:43:11.736947 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.736923    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 13:43:11.736947 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.736928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-c99cs\""
Sep 16 13:43:11.737114 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.736924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 13:43:11.737114 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.736950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 13:43:11.743812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.743790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:43:11.877171 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.877125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdsj\" (UniqueName: \"kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.877354 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.877182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.977655 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.977619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdsj\" (UniqueName: \"kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.977839 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.977673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.980276 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.980229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:11.986377 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:11.986357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdsj\" (UniqueName: \"kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:12.044300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:12.044268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:12.321231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:12.321083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:43:12.326298 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:43:12.326269    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dfb106_b977_4d7e_a857_241bb5905d44.slice/crio-2d1d371dabbf0f0108342a0972d6f5bb8f79e6f344aebfa0adaf719643443f43 WatchSource:0}: Error finding container 2d1d371dabbf0f0108342a0972d6f5bb8f79e6f344aebfa0adaf719643443f43: Status 404 returned error can't find the container with id 2d1d371dabbf0f0108342a0972d6f5bb8f79e6f344aebfa0adaf719643443f43
Sep 16 13:43:12.698256 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:12.698221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" event={"ID":"e4dfb106-b977-4d7e-a857-241bb5905d44","Type":"ContainerStarted","Data":"ed627c7dbb44f9cb9622b1bef4843613185f9958711e190c3a684c67d9a4c86a"}
Sep 16 13:43:12.698435 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:12.698277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" event={"ID":"e4dfb106-b977-4d7e-a857-241bb5905d44","Type":"ContainerStarted","Data":"2d1d371dabbf0f0108342a0972d6f5bb8f79e6f344aebfa0adaf719643443f43"}
Sep 16 13:43:12.721303 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:12.721248    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podStartSLOduration=1.721231039 podStartE2EDuration="1.721231039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:12.720130763 +0000 UTC m=+3298.758929887" watchObservedRunningTime="2026-09-16 13:43:12.721231039 +0000 UTC m=+3298.760030153"
Sep 16 13:43:13.702460 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:13.702428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:14.594618 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:14.594581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:43:14.602549 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:14.602523    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:43:17.706154 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.706115    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:43:17.710835 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.710813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.713479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 13:43:17.713630 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-8mbht\""
Sep 16 13:43:17.713706 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:43:17.713706 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 13:43:17.713889 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 13:43:17.713950 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.713922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 13:43:17.722387 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.722362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:43:17.827763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.827763 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpxz\" (UniqueName: \"kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.828056 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.827984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.928865 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.928818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.928937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.928968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpxz\" (UniqueName: \"kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929496 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929591 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929743 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.929826 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.929748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.931933 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.931902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.932921 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.932895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:17.955096 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:17.955065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpxz\" (UniqueName: \"kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:18.022900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:18.022783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:18.307404 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:18.307322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:43:18.310280 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:43:18.310254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded90e09_2df6_4640_8b85_2fc431e28244.slice/crio-7b4fdf9c52b0dfd12c026ffab78fd41a026c3f572ae3228d8193de7f6bea642d WatchSource:0}: Error finding container 7b4fdf9c52b0dfd12c026ffab78fd41a026c3f572ae3228d8193de7f6bea642d: Status 404 returned error can't find the container with id 7b4fdf9c52b0dfd12c026ffab78fd41a026c3f572ae3228d8193de7f6bea642d
Sep 16 13:43:18.726109 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:18.726072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerStarted","Data":"a0ffef0a27193d238222a6daa57a18031971eaa5126b357f080fdf224ea6b4e6"}
Sep 16 13:43:18.726495 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:18.726129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerStarted","Data":"7b4fdf9c52b0dfd12c026ffab78fd41a026c3f572ae3228d8193de7f6bea642d"}
Sep 16 13:43:21.741582 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:21.741530    2579 generic.go:356] "Generic (PLEG): container finished" podID="ded90e09-2df6-4640-8b85-2fc431e28244" containerID="a0ffef0a27193d238222a6daa57a18031971eaa5126b357f080fdf224ea6b4e6" exitCode=0
Sep 16 13:43:21.741999 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:21.741606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerDied","Data":"a0ffef0a27193d238222a6daa57a18031971eaa5126b357f080fdf224ea6b4e6"}
Sep 16 13:43:22.748404 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:22.748370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerStarted","Data":"d9a2679ec0cb4fc184382aedbaa23190a884b804791fb18da19a3e28954f79ce"}
Sep 16 13:43:22.776145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:22.776095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podStartSLOduration=2.997409552 podStartE2EDuration="5.776076728s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.778667176s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:22.774276955 +0000 UTC m=+3308.813076077" watchObservedRunningTime="2026-09-16 13:43:22.776076728 +0000 UTC m=+3308.814875853"
Sep 16 13:43:23.023793 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:23.023705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:28.023306 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:28.023256    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:43:28.025011 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:28.024980    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:43:38.023913 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:38.023813    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:43:44.708157 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:44.708124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:43:48.023610 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:48.023527    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:43:58.023443 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:43:58.023399    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:44:08.024378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:08.024331    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:44:18.023897 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:18.023847    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:44:28.023750 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:28.023701    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:44:38.033577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:38.033536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:44:38.041505 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:38.041477    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:44:39.615608 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:39.615540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:44:54.638814 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:54.638760    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" containerID="cri-o://d9a2679ec0cb4fc184382aedbaa23190a884b804791fb18da19a3e28954f79ce" gracePeriod=45
Sep 16 13:44:55.034958 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.034921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 16 13:44:55.166524 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.166492    2579 generic.go:356] "Generic (PLEG): container finished" podID="ded90e09-2df6-4640-8b85-2fc431e28244" containerID="d9a2679ec0cb4fc184382aedbaa23190a884b804791fb18da19a3e28954f79ce" exitCode=0
Sep 16 13:44:55.166708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.166572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerDied","Data":"d9a2679ec0cb4fc184382aedbaa23190a884b804791fb18da19a3e28954f79ce"}
Sep 16 13:44:55.187020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.186996    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:44:55.250034 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250198 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250538 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdpxz\" (UniqueName: \"kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz\") pod \"ded90e09-2df6-4640-8b85-2fc431e28244\" (UID: \"ded90e09-2df6-4640-8b85-2fc431e28244\") "
Sep 16 13:44:55.250538 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home" (OuterVolumeSpecName: "home") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:55.250689 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:44:55.250746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:55.250746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.250734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache" (OuterVolumeSpecName: "model-cache") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:55.252682 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.252641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm" (OuterVolumeSpecName: "dshm") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:55.252800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.252686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:44:55.252800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.252733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz" (OuterVolumeSpecName: "kube-api-access-hdpxz") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "kube-api-access-hdpxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:55.278768 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.278708    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ded90e09-2df6-4640-8b85-2fc431e28244" (UID: "ded90e09-2df6-4640-8b85-2fc431e28244"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:55.351891 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351867    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.351891 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351890    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351900    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351909    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351918    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded90e09-2df6-4640-8b85-2fc431e28244-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351926    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ded90e09-2df6-4640-8b85-2fc431e28244-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351933    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ded90e09-2df6-4640-8b85-2fc431e28244-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:55.352044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:55.351947    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdpxz\" (UniqueName: \"kubernetes.io/projected/ded90e09-2df6-4640-8b85-2fc431e28244-kube-api-access-hdpxz\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:44:56.173588 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.173548    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"
Sep 16 13:44:56.174030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.173547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq" event={"ID":"ded90e09-2df6-4640-8b85-2fc431e28244","Type":"ContainerDied","Data":"7b4fdf9c52b0dfd12c026ffab78fd41a026c3f572ae3228d8193de7f6bea642d"}
Sep 16 13:44:56.174030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.173709    2579 scope.go:117] "RemoveContainer" containerID="d9a2679ec0cb4fc184382aedbaa23190a884b804791fb18da19a3e28954f79ce"
Sep 16 13:44:56.185163 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.185142    2579 scope.go:117] "RemoveContainer" containerID="a0ffef0a27193d238222a6daa57a18031971eaa5126b357f080fdf224ea6b4e6"
Sep 16 13:44:56.212136 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.212107    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:44:56.220452 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.220431    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-68cd6868bc-krjdq"]
Sep 16 13:44:56.457449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:44:56.457367    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" path="/var/lib/kubelet/pods/ded90e09-2df6-4640-8b85-2fc431e28244/volumes"
Sep 16 13:45:16.286664 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:16.286627    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:45:24.708293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:24.708249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:31.305702 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:31.305658    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main" containerID="cri-o://ed627c7dbb44f9cb9622b1bef4843613185f9958711e190c3a684c67d9a4c86a" gracePeriod=45
Sep 16 13:45:34.708652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:34.708604    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:44.709035 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:44.708993    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:44.709404 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:44.709095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:45:45.879482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879441    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:45:45.879910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879769    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main"
Sep 16 13:45:45.879910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879783    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main"
Sep 16 13:45:45.879910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="storage-initializer"
Sep 16 13:45:45.879910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="storage-initializer"
Sep 16 13:45:45.879910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.879853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded90e09-2df6-4640-8b85-2fc431e28244" containerName="main"
Sep 16 13:45:45.881839 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.881812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:45.884431 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.884405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-gdqlk\""
Sep 16 13:45:45.884540 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.884419    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 13:45:45.885741 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.885720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 13:45:45.885864 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.885775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-jxglr\""
Sep 16 13:45:45.885864 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.885805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 13:45:45.897132 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.897095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:45:45.979023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.978975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkpjb\" (UniqueName: \"kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:45.979230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:45.979057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.080500 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.080450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkpjb\" (UniqueName: \"kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.080708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.080608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.083183 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.083157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.089020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.088997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkpjb\" (UniqueName: \"kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.193087 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.193051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:46.470711 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:46.470601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:45:46.474050 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:45:46.474008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c765af_884c_484f_a094_fa3df755b0f5.slice/crio-3311b3f1b58c24acaf07eb84390748b9e106f8e60b8013bb71e4dccfc4fd8f15 WatchSource:0}: Error finding container 3311b3f1b58c24acaf07eb84390748b9e106f8e60b8013bb71e4dccfc4fd8f15: Status 404 returned error can't find the container with id 3311b3f1b58c24acaf07eb84390748b9e106f8e60b8013bb71e4dccfc4fd8f15
Sep 16 13:45:47.389330 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:47.389290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" event={"ID":"09c765af-884c-484f-a094-fa3df755b0f5","Type":"ContainerStarted","Data":"4db889126f403221d27e8fa69e0582593a768208167bebbe08560d57afb057c8"}
Sep 16 13:45:47.389767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:47.389351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" event={"ID":"09c765af-884c-484f-a094-fa3df755b0f5","Type":"ContainerStarted","Data":"3311b3f1b58c24acaf07eb84390748b9e106f8e60b8013bb71e4dccfc4fd8f15"}
Sep 16 13:45:47.389767 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:47.389379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:45:47.424266 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:47.424209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" podStartSLOduration=2.424190093 podStartE2EDuration="2.424190093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:47.42217274 +0000 UTC m=+3453.460971864" watchObservedRunningTime="2026-09-16 13:45:47.424190093 +0000 UTC m=+3453.462989215"
Sep 16 13:45:47.969003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:47.968962    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:45:48.394722 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:48.394624    2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" secret="" err="secret \"e2e-del-blocked-svc-epp-sa-dockercfg-gdqlk\" not found"
Sep 16 13:45:48.501785 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:48.501748    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:48.502012 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:48.501834    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:45:49.001810455 +0000 UTC m=+3455.040609557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:49.006470 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:49.006422    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:49.006694 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:49.006510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:45:50.006489877 +0000 UTC m=+3456.045288981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:50.018096 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:50.018056    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:50.018480 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:50.018127    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:45:52.018112943 +0000 UTC m=+3458.056912043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:52.036723 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:52.036677    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:52.037113 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:52.036766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:45:56.036749087 +0000 UTC m=+3462.075548188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:54.708726 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:45:54.708685    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:56.068201 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:56.068156    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:45:56.068648 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:45:56.068235    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:46:04.068218815 +0000 UTC m=+3470.107017915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:46:01.452224 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.452184    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerID="ed627c7dbb44f9cb9622b1bef4843613185f9958711e190c3a684c67d9a4c86a" exitCode=0
Sep 16 13:46:01.452624 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.452250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" event={"ID":"e4dfb106-b977-4d7e-a857-241bb5905d44","Type":"ContainerDied","Data":"ed627c7dbb44f9cb9622b1bef4843613185f9958711e190c3a684c67d9a4c86a"}
Sep 16 13:46:01.553749 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.553723    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:46:01.714624 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.714587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sdsj\" (UniqueName: \"kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj\") pod \"e4dfb106-b977-4d7e-a857-241bb5905d44\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") "
Sep 16 13:46:01.714624 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.714626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs\") pod \"e4dfb106-b977-4d7e-a857-241bb5905d44\" (UID: \"e4dfb106-b977-4d7e-a857-241bb5905d44\") "
Sep 16 13:46:01.716885 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.716851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj" (OuterVolumeSpecName: "kube-api-access-7sdsj") pod "e4dfb106-b977-4d7e-a857-241bb5905d44" (UID: "e4dfb106-b977-4d7e-a857-241bb5905d44"). InnerVolumeSpecName "kube-api-access-7sdsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:46:01.717003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.716963    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4dfb106-b977-4d7e-a857-241bb5905d44" (UID: "e4dfb106-b977-4d7e-a857-241bb5905d44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:01.815385 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.815342    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdsj\" (UniqueName: \"kubernetes.io/projected/e4dfb106-b977-4d7e-a857-241bb5905d44-kube-api-access-7sdsj\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:46:01.815385 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:01.815380    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfb106-b977-4d7e-a857-241bb5905d44-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:46:02.458981 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:02.458940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc" event={"ID":"e4dfb106-b977-4d7e-a857-241bb5905d44","Type":"ContainerDied","Data":"2d1d371dabbf0f0108342a0972d6f5bb8f79e6f344aebfa0adaf719643443f43"}
Sep 16 13:46:02.459554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:02.459011    2579 scope.go:117] "RemoveContainer" containerID="ed627c7dbb44f9cb9622b1bef4843613185f9958711e190c3a684c67d9a4c86a"
Sep 16 13:46:02.459554 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:02.458949    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"
Sep 16 13:46:02.489463 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:02.489429    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:46:02.506881 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:02.506848    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq76mc"]
Sep 16 13:46:04.136363 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:46:04.136311    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:46:04.136809 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:46:04.136400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:46:20.136384326 +0000 UTC m=+3486.175183426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:46:04.418345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:04.418305    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main" containerID="cri-o://4db889126f403221d27e8fa69e0582593a768208167bebbe08560d57afb057c8" gracePeriod=45
Sep 16 13:46:04.459103 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:04.459067    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" path="/var/lib/kubelet/pods/e4dfb106-b977-4d7e-a857-241bb5905d44/volumes"
Sep 16 13:46:17.786715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.786676    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:46:17.787083 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.786974    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main"
Sep 16 13:46:17.787083 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.786986    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main"
Sep 16 13:46:17.787083 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.787038    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dfb106-b977-4d7e-a857-241bb5905d44" containerName="main"
Sep 16 13:46:17.790307 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.790284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:17.793304 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.793283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 13:46:17.794598 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.794554    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-fr6hv\""
Sep 16 13:46:17.794715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.794593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-5v95b\""
Sep 16 13:46:17.794715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.794636    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 16 13:46:17.794715 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.794554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 13:46:17.802001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.801975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:46:17.952040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.951998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:17.952219 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:17.952059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.052669 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.052554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.052669 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.052658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.055337 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.055308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.061949 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.061928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.101676 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.101643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.377327 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.377232    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:46:18.381510 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:46:18.381481    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a73a837_eeb6_4286_8a0e_c013a01d6e48.slice/crio-39d3b1c944da3d8c396f5968caf212b3c13b70963a7f4fa36ea514601e6e0d57 WatchSource:0}: Error finding container 39d3b1c944da3d8c396f5968caf212b3c13b70963a7f4fa36ea514601e6e0d57: Status 404 returned error can't find the container with id 39d3b1c944da3d8c396f5968caf212b3c13b70963a7f4fa36ea514601e6e0d57
Sep 16 13:46:18.532737 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.532696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" event={"ID":"4a73a837-eeb6-4286-8a0e-c013a01d6e48","Type":"ContainerStarted","Data":"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"}
Sep 16 13:46:18.532974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.532763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" event={"ID":"4a73a837-eeb6-4286-8a0e-c013a01d6e48","Type":"ContainerStarted","Data":"39d3b1c944da3d8c396f5968caf212b3c13b70963a7f4fa36ea514601e6e0d57"}
Sep 16 13:46:18.532974 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.532805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:46:18.554967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:18.554915    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" podStartSLOduration=1.554896851 podStartE2EDuration="1.554896851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:18.553601745 +0000 UTC m=+3484.592400891" watchObservedRunningTime="2026-09-16 13:46:18.554896851 +0000 UTC m=+3484.593695975"
Sep 16 13:46:19.403310 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:19.403273    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:46:20.172579 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:46:20.172538    2579 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:46:20.172749 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:46:20.172628    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs podName:09c765af-884c-484f-a094-fa3df755b0f5 nodeName:}" failed. No retries permitted until 2026-09-16 13:46:52.17261457 +0000 UTC m=+3518.211413671 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" (UID: "09c765af-884c-484f-a094-fa3df755b0f5") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 16 13:46:28.328263 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.328226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:46:28.331275 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.331252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.333856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.333827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 13:46:28.335033 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.335007    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 13:46:28.335033 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.335007    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 13:46:28.335228 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.335113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-6xmtf\""
Sep 16 13:46:28.335329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.335313    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-sv6zb\""
Sep 16 13:46:28.343404 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.343377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:46:28.442606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.442550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.442814 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.442709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw95\" (UniqueName: \"kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.543921 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.543890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw95\" (UniqueName: \"kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.544124 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.543987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.546481 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.546456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.552417 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.552393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw95\" (UniqueName: \"kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.645166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.645138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:28.915812 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:28.915780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:46:28.918319 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:46:28.918291    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1921d4e_ef44_4155_a1ff_601c974ee28a.slice/crio-21f20d37ce9d50ee3caebf95b222cc927395f1ed40a9af09e2a2295bcc06bfaf WatchSource:0}: Error finding container 21f20d37ce9d50ee3caebf95b222cc927395f1ed40a9af09e2a2295bcc06bfaf: Status 404 returned error can't find the container with id 21f20d37ce9d50ee3caebf95b222cc927395f1ed40a9af09e2a2295bcc06bfaf
Sep 16 13:46:29.402064 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:29.402028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:46:29.579995 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:29.579949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" event={"ID":"c1921d4e-ef44-4155-a1ff-601c974ee28a","Type":"ContainerStarted","Data":"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"}
Sep 16 13:46:29.580188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:29.580011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" event={"ID":"c1921d4e-ef44-4155-a1ff-601c974ee28a","Type":"ContainerStarted","Data":"21f20d37ce9d50ee3caebf95b222cc927395f1ed40a9af09e2a2295bcc06bfaf"}
Sep 16 13:46:29.580188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:29.580077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:46:29.597238 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:29.597182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podStartSLOduration=1.597161847 podStartE2EDuration="1.597161847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:29.596505444 +0000 UTC m=+3495.635304568" watchObservedRunningTime="2026-09-16 13:46:29.597161847 +0000 UTC m=+3495.635960970"
Sep 16 13:46:30.903113 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:30.903078    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:46:34.604824 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.604794    2579 generic.go:356] "Generic (PLEG): container finished" podID="09c765af-884c-484f-a094-fa3df755b0f5" containerID="4db889126f403221d27e8fa69e0582593a768208167bebbe08560d57afb057c8" exitCode=0
Sep 16 13:46:34.605200 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.604814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" event={"ID":"09c765af-884c-484f-a094-fa3df755b0f5","Type":"ContainerDied","Data":"4db889126f403221d27e8fa69e0582593a768208167bebbe08560d57afb057c8"}
Sep 16 13:46:34.659273 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.659249    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:46:34.696765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.696686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs\") pod \"09c765af-884c-484f-a094-fa3df755b0f5\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") "
Sep 16 13:46:34.696765 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.696744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkpjb\" (UniqueName: \"kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb\") pod \"09c765af-884c-484f-a094-fa3df755b0f5\" (UID: \"09c765af-884c-484f-a094-fa3df755b0f5\") "
Sep 16 13:46:34.698922 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.698894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09c765af-884c-484f-a094-fa3df755b0f5" (UID: "09c765af-884c-484f-a094-fa3df755b0f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:34.699030 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.698953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb" (OuterVolumeSpecName: "kube-api-access-wkpjb") pod "09c765af-884c-484f-a094-fa3df755b0f5" (UID: "09c765af-884c-484f-a094-fa3df755b0f5"). InnerVolumeSpecName "kube-api-access-wkpjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:46:34.797675 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.797632    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkpjb\" (UniqueName: \"kubernetes.io/projected/09c765af-884c-484f-a094-fa3df755b0f5-kube-api-access-wkpjb\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:46:34.797675 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:34.797667    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09c765af-884c-484f-a094-fa3df755b0f5-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:46:35.610851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:35.610819    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"
Sep 16 13:46:35.611321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:35.610818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9" event={"ID":"09c765af-884c-484f-a094-fa3df755b0f5","Type":"ContainerDied","Data":"3311b3f1b58c24acaf07eb84390748b9e106f8e60b8013bb71e4dccfc4fd8f15"}
Sep 16 13:46:35.611321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:35.610973    2579 scope.go:117] "RemoveContainer" containerID="4db889126f403221d27e8fa69e0582593a768208167bebbe08560d57afb057c8"
Sep 16 13:46:35.634012 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:35.633966    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:46:35.636986 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:35.636959    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-c5bb9"]
Sep 16 13:46:36.457910 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:36.457875    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c765af-884c-484f-a094-fa3df755b0f5" path="/var/lib/kubelet/pods/09c765af-884c-484f-a094-fa3df755b0f5/volumes"
Sep 16 13:46:45.922768 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:45.922703    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main" containerID="cri-o://85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43" gracePeriod=45
Sep 16 13:46:47.108454 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.108415    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:46:47.108899 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.108879    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main"
Sep 16 13:46:47.108945 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.108902    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main"
Sep 16 13:46:47.109008 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.108996    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c765af-884c-484f-a094-fa3df755b0f5" containerName="main"
Sep 16 13:46:47.113629 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.113606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.121682 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.121650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 13:46:47.121823 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.121707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-gxk6w\""
Sep 16 13:46:47.121823 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.121746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-f85r9\""
Sep 16 13:46:47.121823 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.121794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 13:46:47.121985 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.121878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 13:46:47.123314 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.123290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:46:47.203345 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.203304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c6k\" (UniqueName: \"kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.203492 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.203384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.304499 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.304463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.304681 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.304537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c6k\" (UniqueName: \"kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.307134 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.307111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.315642 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.315616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c6k\" (UniqueName: \"kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.424119 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.424085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:47.696648 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:47.696620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:46:47.698723 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:46:47.698694    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9011ad0_f47d_44f8_ac90_86b6f7bb7181.slice/crio-c80587675af21e9865317a4feda110dd24e1b299211d5a327eb3decce5d09045 WatchSource:0}: Error finding container c80587675af21e9865317a4feda110dd24e1b299211d5a327eb3decce5d09045: Status 404 returned error can't find the container with id c80587675af21e9865317a4feda110dd24e1b299211d5a327eb3decce5d09045
Sep 16 13:46:48.667611 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:48.667555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" event={"ID":"e9011ad0-f47d-44f8-ac90-86b6f7bb7181","Type":"ContainerStarted","Data":"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"}
Sep 16 13:46:48.668014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:48.667634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" event={"ID":"e9011ad0-f47d-44f8-ac90-86b6f7bb7181","Type":"ContainerStarted","Data":"c80587675af21e9865317a4feda110dd24e1b299211d5a327eb3decce5d09045"}
Sep 16 13:46:48.668014 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:48.667708    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:46:48.685696 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:48.685645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podStartSLOduration=1.685631653 podStartE2EDuration="1.685631653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:48.683733423 +0000 UTC m=+3514.722532545" watchObservedRunningTime="2026-09-16 13:46:48.685631653 +0000 UTC m=+3514.724430777"
Sep 16 13:46:50.906087 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:46:50.906041    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:47:00.587338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:00.587308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:47:00.905553 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:00.905511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:47:10.905002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:10.904957    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:47:16.163212 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.163182    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:47:16.359079 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.358970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs\") pod \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") "
Sep 16 13:47:16.359079 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.359062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf\") pod \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\" (UID: \"4a73a837-eeb6-4286-8a0e-c013a01d6e48\") "
Sep 16 13:47:16.361380 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.361344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a73a837-eeb6-4286-8a0e-c013a01d6e48" (UID: "4a73a837-eeb6-4286-8a0e-c013a01d6e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:47:16.361507 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.361384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf" (OuterVolumeSpecName: "kube-api-access-wxldf") pod "4a73a837-eeb6-4286-8a0e-c013a01d6e48" (UID: "4a73a837-eeb6-4286-8a0e-c013a01d6e48"). InnerVolumeSpecName "kube-api-access-wxldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:47:16.459577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.459518    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/4a73a837-eeb6-4286-8a0e-c013a01d6e48-kube-api-access-wxldf\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:47:16.459577 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.459545    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a73a837-eeb6-4286-8a0e-c013a01d6e48-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:47:16.784735 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.784695    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerID="85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43" exitCode=0
Sep 16 13:47:16.784951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.784746    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"
Sep 16 13:47:16.784951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.784787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" event={"ID":"4a73a837-eeb6-4286-8a0e-c013a01d6e48","Type":"ContainerDied","Data":"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"}
Sep 16 13:47:16.784951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.784856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj" event={"ID":"4a73a837-eeb6-4286-8a0e-c013a01d6e48","Type":"ContainerDied","Data":"39d3b1c944da3d8c396f5968caf212b3c13b70963a7f4fa36ea514601e6e0d57"}
Sep 16 13:47:16.784951 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.784889    2579 scope.go:117] "RemoveContainer" containerID="85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"
Sep 16 13:47:16.793329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.793309    2579 scope.go:117] "RemoveContainer" containerID="85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"
Sep 16 13:47:16.793654 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:47:16.793635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43\": container with ID starting with 85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43 not found: ID does not exist" containerID="85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"
Sep 16 13:47:16.793711 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.793661    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43"} err="failed to get container status \"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43\": rpc error: code = NotFound desc = could not find container \"85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43\": container with ID starting with 85aa14df4903525e9f473b01890dbe208bebd19e92b3a058793b487920cc1a43 not found: ID does not exist"
Sep 16 13:47:16.802422 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.802390    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:47:16.806308 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:16.806280    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddc8cfj"]
Sep 16 13:47:18.457800 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:18.457767    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" path="/var/lib/kubelet/pods/4a73a837-eeb6-4286-8a0e-c013a01d6e48/volumes"
Sep 16 13:47:19.673346 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:47:19.673315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:48:14.622927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:48:14.622893    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:48:14.629890 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:48:14.629864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:48:55.825020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:48:55.824985    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:49:00.588145 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:00.588098    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:08.917933 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:08.917891    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:49:09.674092 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:09.674050    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:10.587645 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:10.587602    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:10.844484 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:10.844392    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" containerID="cri-o://8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0" gracePeriod=45
Sep 16 13:49:19.673967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:19.673917    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:20.587851 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:20.587811    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:20.588026 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:20.587925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:49:22.266708 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.260074    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:49:22.267366 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.267333    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main"
Sep 16 13:49:22.267366 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.267368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main"
Sep 16 13:49:22.267544 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.267515    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a73a837-eeb6-4286-8a0e-c013a01d6e48" containerName="main"
Sep 16 13:49:22.271044 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.271018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.273751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.273725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-7csv6\""
Sep 16 13:49:22.274680 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.274653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:49:22.371359 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.371318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99bm\" (UniqueName: \"kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.371536 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.371385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.472627 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.472589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99bm\" (UniqueName: \"kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.472829 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.472652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.475116 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.475093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.480507 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.480481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99bm\" (UniqueName: \"kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.583911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.583826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:22.850452 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:22.850421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:49:22.853254 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:49:22.853216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e861bc_ff16_465c_a794_4d8cfd7bf1c6.slice/crio-b2f159af1320b190ba618b1e13cf9791d65d5e02c30701e8e52efe5555c5a837 WatchSource:0}: Error finding container b2f159af1320b190ba618b1e13cf9791d65d5e02c30701e8e52efe5555c5a837: Status 404 returned error can't find the container with id b2f159af1320b190ba618b1e13cf9791d65d5e02c30701e8e52efe5555c5a837
Sep 16 13:49:23.307899 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:23.307864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" event={"ID":"43e861bc-ff16-465c-a794-4d8cfd7bf1c6","Type":"ContainerStarted","Data":"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"}
Sep 16 13:49:23.308347 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:23.307918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" event={"ID":"43e861bc-ff16-465c-a794-4d8cfd7bf1c6","Type":"ContainerStarted","Data":"b2f159af1320b190ba618b1e13cf9791d65d5e02c30701e8e52efe5555c5a837"}
Sep 16 13:49:23.308347 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:23.307943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:23.327407 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:23.327354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podStartSLOduration=1.327339821 podStartE2EDuration="1.327339821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:23.326619865 +0000 UTC m=+3669.365419015" watchObservedRunningTime="2026-09-16 13:49:23.327339821 +0000 UTC m=+3669.366138944"
Sep 16 13:49:23.937677 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:23.937638    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" containerID="cri-o://caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6" gracePeriod=45
Sep 16 13:49:29.674100 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:29.674060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:29.674733 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:29.674165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:49:30.587549 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:30.587508    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:39.674107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:39.674025    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:40.587515 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:40.587476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:41.084398 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.084366    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:49:41.224674 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.224641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcw95\" (UniqueName: \"kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95\") pod \"c1921d4e-ef44-4155-a1ff-601c974ee28a\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") "
Sep 16 13:49:41.224869 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.224689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs\") pod \"c1921d4e-ef44-4155-a1ff-601c974ee28a\" (UID: \"c1921d4e-ef44-4155-a1ff-601c974ee28a\") "
Sep 16 13:49:41.226926 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.226894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1921d4e-ef44-4155-a1ff-601c974ee28a" (UID: "c1921d4e-ef44-4155-a1ff-601c974ee28a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:49:41.226926 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.226908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95" (OuterVolumeSpecName: "kube-api-access-kcw95") pod "c1921d4e-ef44-4155-a1ff-601c974ee28a" (UID: "c1921d4e-ef44-4155-a1ff-601c974ee28a"). InnerVolumeSpecName "kube-api-access-kcw95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:49:41.326202 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.326164    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcw95\" (UniqueName: \"kubernetes.io/projected/c1921d4e-ef44-4155-a1ff-601c974ee28a-kube-api-access-kcw95\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:49:41.326202 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.326200    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1921d4e-ef44-4155-a1ff-601c974ee28a-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:49:41.387158 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.387124    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerID="8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0" exitCode=0
Sep 16 13:49:41.387329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.387172    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"
Sep 16 13:49:41.387329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.387209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" event={"ID":"c1921d4e-ef44-4155-a1ff-601c974ee28a","Type":"ContainerDied","Data":"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"}
Sep 16 13:49:41.387329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.387262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc" event={"ID":"c1921d4e-ef44-4155-a1ff-601c974ee28a","Type":"ContainerDied","Data":"21f20d37ce9d50ee3caebf95b222cc927395f1ed40a9af09e2a2295bcc06bfaf"}
Sep 16 13:49:41.387329 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.387285    2579 scope.go:117] "RemoveContainer" containerID="8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"
Sep 16 13:49:41.396506 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.396488    2579 scope.go:117] "RemoveContainer" containerID="8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"
Sep 16 13:49:41.396819 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:49:41.396798    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0\": container with ID starting with 8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0 not found: ID does not exist" containerID="8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"
Sep 16 13:49:41.396881 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.396827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0"} err="failed to get container status \"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0\": rpc error: code = NotFound desc = could not find container \"8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0\": container with ID starting with 8bb3250e7bde7e678b2db9e902d56f5ff2b1419645af90416432a005e94735d0 not found: ID does not exist"
Sep 16 13:49:41.409719 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.409698    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:49:41.414416 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:41.414394    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc585wxwc"]
Sep 16 13:49:42.458419 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:42.458387    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" path="/var/lib/kubelet/pods/c1921d4e-ef44-4155-a1ff-601c974ee28a/volumes"
Sep 16 13:49:49.673816 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:49.673774    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:49:54.177652 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.177626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:49:54.226341 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.226262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs\") pod \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") "
Sep 16 13:49:54.226341 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.226331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48c6k\" (UniqueName: \"kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k\") pod \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\" (UID: \"e9011ad0-f47d-44f8-ac90-86b6f7bb7181\") "
Sep 16 13:49:54.228601 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.228550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9011ad0-f47d-44f8-ac90-86b6f7bb7181" (UID: "e9011ad0-f47d-44f8-ac90-86b6f7bb7181"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:49:54.228601 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.228585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k" (OuterVolumeSpecName: "kube-api-access-48c6k") pod "e9011ad0-f47d-44f8-ac90-86b6f7bb7181" (UID: "e9011ad0-f47d-44f8-ac90-86b6f7bb7181"). InnerVolumeSpecName "kube-api-access-48c6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:49:54.314456 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.314428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:49:54.327107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.327079    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:49:54.327107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.327103    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48c6k\" (UniqueName: \"kubernetes.io/projected/e9011ad0-f47d-44f8-ac90-86b6f7bb7181-kube-api-access-48c6k\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:49:54.441634 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.441594    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerID="caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6" exitCode=0
Sep 16 13:49:54.441790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.441677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" event={"ID":"e9011ad0-f47d-44f8-ac90-86b6f7bb7181","Type":"ContainerDied","Data":"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"}
Sep 16 13:49:54.441790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.441687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"
Sep 16 13:49:54.441790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.441714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn" event={"ID":"e9011ad0-f47d-44f8-ac90-86b6f7bb7181","Type":"ContainerDied","Data":"c80587675af21e9865317a4feda110dd24e1b299211d5a327eb3decce5d09045"}
Sep 16 13:49:54.441790 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.441735    2579 scope.go:117] "RemoveContainer" containerID="caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"
Sep 16 13:49:54.450020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.449916    2579 scope.go:117] "RemoveContainer" containerID="caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"
Sep 16 13:49:54.450312 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:49:54.450290    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6\": container with ID starting with caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6 not found: ID does not exist" containerID="caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"
Sep 16 13:49:54.450405 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.450318    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6"} err="failed to get container status \"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6\": rpc error: code = NotFound desc = could not find container \"caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6\": container with ID starting with caf7244fb8036fa89184f3be53841de5b8b789bc7d5cc33f9afe9ac7debe9ee6 not found: ID does not exist"
Sep 16 13:49:54.463714 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.463689    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:49:54.470271 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:54.470248    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-ddgzn"]
Sep 16 13:49:56.457380 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:49:56.457343    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" path="/var/lib/kubelet/pods/e9011ad0-f47d-44f8-ac90-86b6f7bb7181/volumes"
Sep 16 13:50:11.765028 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.764996    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765312    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main"
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765324    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main"
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765336    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main"
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765341    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main"
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765401    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9011ad0-f47d-44f8-ac90-86b6f7bb7181" containerName="main"
Sep 16 13:50:11.765497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.765409    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1921d4e-ef44-4155-a1ff-601c974ee28a" containerName="main"
Sep 16 13:50:11.768249 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.768232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.770755 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.770733    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 13:50:11.772074 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.772044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 13:50:11.772074 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.772052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 13:50:11.772272 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.772084    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-ghtcn\""
Sep 16 13:50:11.772378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.772323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-qjmvw\""
Sep 16 13:50:11.776468 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.776447    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:50:11.868903 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.868867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkj4\" (UniqueName: \"kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.869065 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.868930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.970167 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.970133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkj4\" (UniqueName: \"kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.970321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.970193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.972673 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.972652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:11.979027 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:11.979003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkj4\" (UniqueName: \"kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:12.079199 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.079118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:12.349927 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.349898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:50:12.353027 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:50:12.353000    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fb357e_b684_4ede_8c67_c3f99de267d6.slice/crio-4752fe47891443df6cff0446945388907405e6419aa63fc8a81347c77bfab2a3 WatchSource:0}: Error finding container 4752fe47891443df6cff0446945388907405e6419aa63fc8a81347c77bfab2a3: Status 404 returned error can't find the container with id 4752fe47891443df6cff0446945388907405e6419aa63fc8a81347c77bfab2a3
Sep 16 13:50:12.518093 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.518056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" event={"ID":"a8fb357e-b684-4ede-8c67-c3f99de267d6","Type":"ContainerStarted","Data":"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"}
Sep 16 13:50:12.518291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.518108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" event={"ID":"a8fb357e-b684-4ede-8c67-c3f99de267d6","Type":"ContainerStarted","Data":"4752fe47891443df6cff0446945388907405e6419aa63fc8a81347c77bfab2a3"}
Sep 16 13:50:12.518291 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.518180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:50:12.537330 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:12.537286    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podStartSLOduration=1.537271109 podStartE2EDuration="1.537271109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:12.536682599 +0000 UTC m=+3718.575481724" watchObservedRunningTime="2026-09-16 13:50:12.537271109 +0000 UTC m=+3718.576070272"
Sep 16 13:50:43.524838 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:50:43.524804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:51:24.113268 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:24.113223    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:51:24.314967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:24.314922    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:51:34.314620 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:34.314557    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:51:39.133101 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:39.133022    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" containerID="cri-o://f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d" gracePeriod=45
Sep 16 13:51:44.314250 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:44.314212    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:51:44.314680 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:44.314320    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:51:54.314081 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:51:54.314042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:52:04.314860 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:04.314819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:52:09.373243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.373216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:52:09.393121 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.393042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs\") pod \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") "
Sep 16 13:52:09.393121 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.393092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v99bm\" (UniqueName: \"kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm\") pod \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\" (UID: \"43e861bc-ff16-465c-a794-4d8cfd7bf1c6\") "
Sep 16 13:52:09.395380 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.395351    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43e861bc-ff16-465c-a794-4d8cfd7bf1c6" (UID: "43e861bc-ff16-465c-a794-4d8cfd7bf1c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:52:09.395486 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.395454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm" (OuterVolumeSpecName: "kube-api-access-v99bm") pod "43e861bc-ff16-465c-a794-4d8cfd7bf1c6" (UID: "43e861bc-ff16-465c-a794-4d8cfd7bf1c6"). InnerVolumeSpecName "kube-api-access-v99bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:52:09.494055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.494018    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:52:09.494055 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.494051    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v99bm\" (UniqueName: \"kubernetes.io/projected/43e861bc-ff16-465c-a794-4d8cfd7bf1c6-kube-api-access-v99bm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:52:09.998682 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.998644    2579 generic.go:356] "Generic (PLEG): container finished" podID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerID="f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d" exitCode=0
Sep 16 13:52:09.998866 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.998692    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"
Sep 16 13:52:09.998866 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.998730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" event={"ID":"43e861bc-ff16-465c-a794-4d8cfd7bf1c6","Type":"ContainerDied","Data":"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"}
Sep 16 13:52:09.998866 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.998785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w" event={"ID":"43e861bc-ff16-465c-a794-4d8cfd7bf1c6","Type":"ContainerDied","Data":"b2f159af1320b190ba618b1e13cf9791d65d5e02c30701e8e52efe5555c5a837"}
Sep 16 13:52:09.998866 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:09.998807    2579 scope.go:117] "RemoveContainer" containerID="f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"
Sep 16 13:52:10.007902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:10.007881    2579 scope.go:117] "RemoveContainer" containerID="f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"
Sep 16 13:52:10.008206 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:52:10.008186    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d\": container with ID starting with f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d not found: ID does not exist" containerID="f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"
Sep 16 13:52:10.008265 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:10.008213    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d"} err="failed to get container status \"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d\": rpc error: code = NotFound desc = could not find container \"f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d\": container with ID starting with f1c36885cec00fb0e7460b4f21ac0668ef4deed62fce96ce05e869e41b08576d not found: ID does not exist"
Sep 16 13:52:10.020047 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:10.020021    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:52:10.025205 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:10.025183    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-gr95w"]
Sep 16 13:52:10.458166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:10.458124    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" path="/var/lib/kubelet/pods/43e861bc-ff16-465c-a794-4d8cfd7bf1c6/volumes"
Sep 16 13:52:26.766497 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.766457    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:52:26.766978 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.766761    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main"
Sep 16 13:52:26.766978 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.766771    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main"
Sep 16 13:52:26.766978 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.766826    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e861bc-ff16-465c-a794-4d8cfd7bf1c6" containerName="main"
Sep 16 13:52:26.770829 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.770812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.793382 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.793354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 13:52:26.793516 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.793358    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-rg4ks\""
Sep 16 13:52:26.793799 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.793783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 13:52:26.793882 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.793790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 13:52:26.794784 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.794770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:52:26.794874 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.794854    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 13:52:26.807945 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.807904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:52:26.832950 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.832923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.832954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.832989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.833044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.833110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdjq\" (UniqueName: \"kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.833160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.833193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.833253 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.833221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934521 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934521 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdjq\" (UniqueName: \"kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.934967 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.935130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.934962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.935130 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.935052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.935239 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.935209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.935427 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.935398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.936078 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.935531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.937521 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.937491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.937683 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.937660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:26.943341 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:26.943316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdjq\" (UniqueName: \"kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq\") pod \"lora-single-adapter-test-kserve-547c756b66-rhz7n\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:27.080787 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:27.080702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:27.373015 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:27.372914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:52:28.075616 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:28.075554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerStarted","Data":"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"}
Sep 16 13:52:28.076126 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:28.075647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerStarted","Data":"4b9d075520694e4be39592bb8a5d91ecbe8aaf3b3325395e797056453e591366"}
Sep 16 13:52:28.466622 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:28.465482    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:52:31.091474 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:31.091372    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef7c9418-b974-42d3-806a-75e790f27882" containerID="1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094" exitCode=0
Sep 16 13:52:31.091474 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:31.091444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerDied","Data":"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"}
Sep 16 13:52:32.097847 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:32.097814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerStarted","Data":"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"}
Sep 16 13:52:32.123658 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:32.123604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podStartSLOduration=2.901478693 podStartE2EDuration="6.123583019s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.222104326s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:32.12135601 +0000 UTC m=+3858.160155133" watchObservedRunningTime="2026-09-16 13:52:32.123583019 +0000 UTC m=+3858.162382139"
Sep 16 13:52:33.081074 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:33.081035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:33.524507 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:33.524460    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:52:37.081662 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:37.081538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:52:37.083180 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:37.083148    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:52:43.503983 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:43.503914    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" containerID="cri-o://0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95" gracePeriod=45
Sep 16 13:52:43.525095 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:43.525046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:52:47.081286 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:47.081228    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:52:53.524368 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:53.524325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:52:53.524762 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:53.524441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:52:57.082186 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:52:57.082130    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:53:03.524895 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:03.524849    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:53:07.081479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:07.081429    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:53:13.523857 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:53:13.523835    2579 logging.go:55] [core] [Channel #1313 SubChannel #1314]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.77:9003", ServerName: "10.134.0.77:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.77:9003: connect: connection refused"
Sep 16 13:53:13.750315 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.750287    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:53:13.835062 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.834970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs\") pod \"a8fb357e-b684-4ede-8c67-c3f99de267d6\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") "
Sep 16 13:53:13.835231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.835085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twkj4\" (UniqueName: \"kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4\") pod \"a8fb357e-b684-4ede-8c67-c3f99de267d6\" (UID: \"a8fb357e-b684-4ede-8c67-c3f99de267d6\") "
Sep 16 13:53:13.837338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.837307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8fb357e-b684-4ede-8c67-c3f99de267d6" (UID: "a8fb357e-b684-4ede-8c67-c3f99de267d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:53:13.837452 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.837331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4" (OuterVolumeSpecName: "kube-api-access-twkj4") pod "a8fb357e-b684-4ede-8c67-c3f99de267d6" (UID: "a8fb357e-b684-4ede-8c67-c3f99de267d6"). InnerVolumeSpecName "kube-api-access-twkj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:53:13.935937 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.935895    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twkj4\" (UniqueName: \"kubernetes.io/projected/a8fb357e-b684-4ede-8c67-c3f99de267d6-kube-api-access-twkj4\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:53:13.935937 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:13.935930    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8fb357e-b684-4ede-8c67-c3f99de267d6-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:53:14.291089 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.291050    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerID="0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95" exitCode=0
Sep 16 13:53:14.291289 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.291094    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"
Sep 16 13:53:14.291289 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.291099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" event={"ID":"a8fb357e-b684-4ede-8c67-c3f99de267d6","Type":"ContainerDied","Data":"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"}
Sep 16 13:53:14.291289 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.291223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" event={"ID":"a8fb357e-b684-4ede-8c67-c3f99de267d6","Type":"ContainerDied","Data":"4752fe47891443df6cff0446945388907405e6419aa63fc8a81347c77bfab2a3"}
Sep 16 13:53:14.291289 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.291247    2579 scope.go:117] "RemoveContainer" containerID="0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"
Sep 16 13:53:14.300207 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.300188    2579 scope.go:117] "RemoveContainer" containerID="0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"
Sep 16 13:53:14.300532 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:53:14.300509    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95\": container with ID starting with 0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95 not found: ID does not exist" containerID="0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"
Sep 16 13:53:14.300641 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.300543    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95"} err="failed to get container status \"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95\": rpc error: code = NotFound desc = could not find container \"0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95\": container with ID starting with 0eee4faec6a5a7278b660ed8c96a4ed883d5a99a2a9bf736847ed6dc6b1eab95 not found: ID does not exist"
Sep 16 13:53:14.314300 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.314274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:53:14.317639 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.317618    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2"]
Sep 16 13:53:14.459539 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.459503    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" path="/var/lib/kubelet/pods/a8fb357e-b684-4ede-8c67-c3f99de267d6/volumes"
Sep 16 13:53:14.524605 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.524548    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-clnq2" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.77:9003\" within 1s: context deadline exceeded"
Sep 16 13:53:14.524605 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:53:14.524593    2579 logging.go:55] [core] [Channel #1313 SubChannel #1314]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.77:9003", ServerName: "10.134.0.77:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.77:9003: operation was canceled"
Sep 16 13:53:14.657503 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.657470    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:53:14.662146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:14.662123    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:53:17.081923 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:17.081876    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:53:26.023654 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.023618    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:53:26.024021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.023899    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main"
Sep 16 13:53:26.024021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.023911    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main"
Sep 16 13:53:26.024021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.023969    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fb357e-b684-4ede-8c67-c3f99de267d6" containerName="main"
Sep 16 13:53:26.027551 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.027532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.033194 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.033164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 13:53:26.034165 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.034021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 13:53:26.034813 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.034791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-w77p2\""
Sep 16 13:53:26.034911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.034820    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 13:53:26.034911 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.034875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-5pv5w\""
Sep 16 13:53:26.051143 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.051114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:53:26.144947 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.144894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgcl\" (UniqueName: \"kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.145166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.145024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.245955 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.245917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgcl\" (UniqueName: \"kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.246165 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.246009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.248658 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.248628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.259339 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.259308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgcl\" (UniqueName: \"kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.338277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.338188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:26.617630 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:26.613143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:53:26.620357 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:53:26.620020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083a0ef3_5758_4b40_a8b0_661848efb60d.slice/crio-c7d92e9b703fd87214da7c13cf14e78505a702c93e342d925c4f95478dcefe85 WatchSource:0}: Error finding container c7d92e9b703fd87214da7c13cf14e78505a702c93e342d925c4f95478dcefe85: Status 404 returned error can't find the container with id c7d92e9b703fd87214da7c13cf14e78505a702c93e342d925c4f95478dcefe85
Sep 16 13:53:27.081509 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:27.081462    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:53:27.353146 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:27.353103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" event={"ID":"083a0ef3-5758-4b40-a8b0-661848efb60d","Type":"ContainerStarted","Data":"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"}
Sep 16 13:53:27.353321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:27.353163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" event={"ID":"083a0ef3-5758-4b40-a8b0-661848efb60d","Type":"ContainerStarted","Data":"c7d92e9b703fd87214da7c13cf14e78505a702c93e342d925c4f95478dcefe85"}
Sep 16 13:53:27.353321 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:27.353263    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:53:37.081277 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:37.081227    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 16 13:53:47.091407 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:47.091372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:53:47.099742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:47.099708    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:53:47.128614 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:47.128540    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podStartSLOduration=22.128523625 podStartE2EDuration="22.128523625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:27.390945687 +0000 UTC m=+3913.429744810" watchObservedRunningTime="2026-09-16 13:53:47.128523625 +0000 UTC m=+3933.167322747"
Sep 16 13:53:58.359006 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:53:58.358970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:54:12.948216 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:12.948112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-547c756b66-rhz7n_ef7c9418-b974-42d3-806a-75e790f27882/storage-initializer/0.log"
Sep 16 13:54:12.956448 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:12.956407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-547c756b66-rhz7n_ef7c9418-b974-42d3-806a-75e790f27882/main/0.log"
Sep 16 13:54:14.328606 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:14.328554    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:54:29.351207 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:29.351156    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main" containerID="cri-o://a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610" gracePeriod=45
Sep 16 13:54:30.010538 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.010509    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:54:30.098723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmdjq\" (UniqueName: \"kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.098960 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.098894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home\") pod \"ef7c9418-b974-42d3-806a-75e790f27882\" (UID: \"ef7c9418-b974-42d3-806a-75e790f27882\") "
Sep 16 13:54:30.099313 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.099261    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:30.099383 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.099339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:54:30.099514 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.099479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home" (OuterVolumeSpecName: "home") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:30.101166 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.101138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:54:30.101338 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.101309    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm" (OuterVolumeSpecName: "dshm") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:30.101594 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.101546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq" (OuterVolumeSpecName: "kube-api-access-lmdjq") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "kube-api-access-lmdjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:30.116434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.116409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:30.123225 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.123202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef7c9418-b974-42d3-806a-75e790f27882" (UID: "ef7c9418-b974-42d3-806a-75e790f27882"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:30.199751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199718    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199747    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7c9418-b974-42d3-806a-75e790f27882-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199751 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199757    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199765    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199774    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199784    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdjq\" (UniqueName: \"kubernetes.io/projected/ef7c9418-b974-42d3-806a-75e790f27882-kube-api-access-lmdjq\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199792    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef7c9418-b974-42d3-806a-75e790f27882-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.199968 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.199802    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7c9418-b974-42d3-806a-75e790f27882-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:54:30.629261 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.629223    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef7c9418-b974-42d3-806a-75e790f27882" containerID="a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610" exitCode=0
Sep 16 13:54:30.629722 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.629285    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"
Sep 16 13:54:30.629722 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.629288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerDied","Data":"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"}
Sep 16 13:54:30.629722 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.629401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n" event={"ID":"ef7c9418-b974-42d3-806a-75e790f27882","Type":"ContainerDied","Data":"4b9d075520694e4be39592bb8a5d91ecbe8aaf3b3325395e797056453e591366"}
Sep 16 13:54:30.629722 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.629422    2579 scope.go:117] "RemoveContainer" containerID="a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"
Sep 16 13:54:30.637677 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.637650    2579 scope.go:117] "RemoveContainer" containerID="1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"
Sep 16 13:54:30.645297 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.645273    2579 scope.go:117] "RemoveContainer" containerID="a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"
Sep 16 13:54:30.645614 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:54:30.645588    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610\": container with ID starting with a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610 not found: ID does not exist" containerID="a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"
Sep 16 13:54:30.645680 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.645622    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610"} err="failed to get container status \"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610\": rpc error: code = NotFound desc = could not find container \"a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610\": container with ID starting with a6a809d73edfd98834355fc2f563f873abac657cd03e506fa33e567d79226610 not found: ID does not exist"
Sep 16 13:54:30.645680 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.645639    2579 scope.go:117] "RemoveContainer" containerID="1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"
Sep 16 13:54:30.645966 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:54:30.645939    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094\": container with ID starting with 1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094 not found: ID does not exist" containerID="1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"
Sep 16 13:54:30.646036 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.645990    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094"} err="failed to get container status \"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094\": rpc error: code = NotFound desc = could not find container \"1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094\": container with ID starting with 1da83085f56b294e281b560f46a7cfbd8cffe5809df6810a0f7ca3f8cc845094 not found: ID does not exist"
Sep 16 13:54:30.647298 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.647254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:54:30.650280 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:30.650259    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-547c756b66-rhz7n"]
Sep 16 13:54:32.463848 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:32.463813    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7c9418-b974-42d3-806a-75e790f27882" path="/var/lib/kubelet/pods/ef7c9418-b974-42d3-806a-75e790f27882/volumes"
Sep 16 13:54:47.128322 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128285    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:54:47.128843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128769    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="storage-initializer"
Sep 16 13:54:47.128843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128789    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="storage-initializer"
Sep 16 13:54:47.128843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128800    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main"
Sep 16 13:54:47.128843 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128808    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main"
Sep 16 13:54:47.129053 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.128888    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7c9418-b974-42d3-806a-75e790f27882" containerName="main"
Sep 16 13:54:47.131020 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.130997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.133699 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.133673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 13:54:47.133830 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.133815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 13:54:47.134068 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.134052    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 13:54:47.134401 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.134384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 13:54:47.134474 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.134455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-v6sm4\""
Sep 16 13:54:47.134836 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.134820    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:54:47.157183 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.157150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:54:47.241483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241483 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbk2\" (UniqueName: \"kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.241742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.241724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342333 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342333 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342333 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342333 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbk2\" (UniqueName: \"kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342537 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342537 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342693 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342757 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342810 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.342918 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.343038 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.342938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.344714 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.344693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.345150 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.345131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.350605 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.350580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbk2\" (UniqueName: \"kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2\") pod \"lora-multiple-adapters-test-kserve-6664897865-jvmxj\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.442401 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.442360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:47.727611 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:47.727536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:54:47.730024 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:54:47.729988    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629d8529_625b_4889_8c4d_4b13222cc25c.slice/crio-7d49b4f5ba75022215e6f462908b2395db61e7d34b9481c855e170e4091ca017 WatchSource:0}: Error finding container 7d49b4f5ba75022215e6f462908b2395db61e7d34b9481c855e170e4091ca017: Status 404 returned error can't find the container with id 7d49b4f5ba75022215e6f462908b2395db61e7d34b9481c855e170e4091ca017
Sep 16 13:54:48.706256 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:48.706209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerStarted","Data":"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"}
Sep 16 13:54:48.706757 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:48.706281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerStarted","Data":"7d49b4f5ba75022215e6f462908b2395db61e7d34b9481c855e170e4091ca017"}
Sep 16 13:54:51.720449 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:51.720411    2579 generic.go:356] "Generic (PLEG): container finished" podID="629d8529-625b-4889-8c4d-4b13222cc25c" containerID="929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6" exitCode=0
Sep 16 13:54:51.720863 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:51.720482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerDied","Data":"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"}
Sep 16 13:54:52.727413 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:52.727369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerStarted","Data":"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"}
Sep 16 13:54:52.753637 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:52.753510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podStartSLOduration=2.6415391599999998 podStartE2EDuration="5.753487546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.111948386s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:54:52.752742855 +0000 UTC m=+3998.791541993" watchObservedRunningTime="2026-09-16 13:54:52.753487546 +0000 UTC m=+3998.792286672"
Sep 16 13:54:53.442964 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:53.442915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:57.443534 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:57.443496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:54:57.445247 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:54:57.445212    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:07.443278 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:07.443223    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:17.443627 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:17.443537    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:27.443262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:27.443213    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:37.443753 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:37.443649    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:47.443547 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:47.443497    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:55:57.443926 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:55:57.443878    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:56:07.442935 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:07.442885    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 16 13:56:15.324991 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:15.324952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:56:17.453372 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:17.453329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:56:17.461896 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:17.461863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:56:18.359038 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:18.358996    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:56:28.358828 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:28.358777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:56:30.344430 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:30.344352    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" containerID="cri-o://adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7" gracePeriod=45
Sep 16 13:56:38.359737 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:38.359700    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:56:38.360222 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:38.359811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:56:46.529050 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:46.529016    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6664897865-jvmxj_629d8529-625b-4889-8c4d-4b13222cc25c/storage-initializer/0.log"
Sep 16 13:56:46.539105 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:46.539082    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6664897865-jvmxj_629d8529-625b-4889-8c4d-4b13222cc25c/main/0.log"
Sep 16 13:56:47.483139 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:47.483100    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:56:48.359578 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:48.359527    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:56:58.359378 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:56:58.359335    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:57:00.591887 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.591855    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:57:00.705023 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.704984    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdgcl\" (UniqueName: \"kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl\") pod \"083a0ef3-5758-4b40-a8b0-661848efb60d\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") "
Sep 16 13:57:00.705230 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.705102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs\") pod \"083a0ef3-5758-4b40-a8b0-661848efb60d\" (UID: \"083a0ef3-5758-4b40-a8b0-661848efb60d\") "
Sep 16 13:57:00.707290 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.707255    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "083a0ef3-5758-4b40-a8b0-661848efb60d" (UID: "083a0ef3-5758-4b40-a8b0-661848efb60d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:00.707402 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.707284    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl" (OuterVolumeSpecName: "kube-api-access-qdgcl") pod "083a0ef3-5758-4b40-a8b0-661848efb60d" (UID: "083a0ef3-5758-4b40-a8b0-661848efb60d"). InnerVolumeSpecName "kube-api-access-qdgcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:00.806243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.806182    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgcl\" (UniqueName: \"kubernetes.io/projected/083a0ef3-5758-4b40-a8b0-661848efb60d-kube-api-access-qdgcl\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:00.806243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:00.806236    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083a0ef3-5758-4b40-a8b0-661848efb60d-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:01.308392 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.308353    2579 generic.go:356] "Generic (PLEG): container finished" podID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerID="adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7" exitCode=0
Sep 16 13:57:01.308628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.308400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"
Sep 16 13:57:01.308628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.308437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" event={"ID":"083a0ef3-5758-4b40-a8b0-661848efb60d","Type":"ContainerDied","Data":"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"}
Sep 16 13:57:01.308628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.308490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7" event={"ID":"083a0ef3-5758-4b40-a8b0-661848efb60d","Type":"ContainerDied","Data":"c7d92e9b703fd87214da7c13cf14e78505a702c93e342d925c4f95478dcefe85"}
Sep 16 13:57:01.308628 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.308513    2579 scope.go:117] "RemoveContainer" containerID="adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"
Sep 16 13:57:01.317912 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.317892    2579 scope.go:117] "RemoveContainer" containerID="adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"
Sep 16 13:57:01.318312 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:57:01.318287    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7\": container with ID starting with adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7 not found: ID does not exist" containerID="adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"
Sep 16 13:57:01.318423 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.318318    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7"} err="failed to get container status \"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7\": rpc error: code = NotFound desc = could not find container \"adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7\": container with ID starting with adfcedcc6157873a7bdf0ec1ad6ff7bcb6aefe37d045327a20a50e558dbb53c7 not found: ID does not exist"
Sep 16 13:57:01.331672 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.331637    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:57:01.337689 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:01.337659    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schejthl7"]
Sep 16 13:57:02.457479 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:02.457439    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" path="/var/lib/kubelet/pods/083a0ef3-5758-4b40-a8b0-661848efb60d/volumes"
Sep 16 13:57:02.506670 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:02.506627    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main" containerID="cri-o://d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0" gracePeriod=45
Sep 16 13:57:03.159805 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.159778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:57:03.319243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.319151    2579 generic.go:356] "Generic (PLEG): container finished" podID="629d8529-625b-4889-8c4d-4b13222cc25c" containerID="d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0" exitCode=0
Sep 16 13:57:03.319243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.319201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerDied","Data":"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"}
Sep 16 13:57:03.319243 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.319240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj" event={"ID":"629d8529-625b-4889-8c4d-4b13222cc25c","Type":"ContainerDied","Data":"7d49b4f5ba75022215e6f462908b2395db61e7d34b9481c855e170e4091ca017"}
Sep 16 13:57:03.319492 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.319264    2579 scope.go:117] "RemoveContainer" containerID="d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"
Sep 16 13:57:03.319492 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.319204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"
Sep 16 13:57:03.328809 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.328975 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.328975 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.328975 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klbk2\" (UniqueName: \"kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.328975 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.328975 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.328965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.329231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.329022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.329231 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.329048    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location\") pod \"629d8529-625b-4889-8c4d-4b13222cc25c\" (UID: \"629d8529-625b-4889-8c4d-4b13222cc25c\") "
Sep 16 13:57:03.329340 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.329221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home" (OuterVolumeSpecName: "home") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:03.329533 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.329513    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.329899 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.329863    2579 scope.go:117] "RemoveContainer" containerID="929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"
Sep 16 13:57:03.330257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.330065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache" (OuterVolumeSpecName: "model-cache") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:03.330257 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.330228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:57:03.331902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.331832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm" (OuterVolumeSpecName: "dshm") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:03.331902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.331851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:03.332188 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.332168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2" (OuterVolumeSpecName: "kube-api-access-klbk2") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "kube-api-access-klbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:03.341454 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.341411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:03.347872 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.347850    2579 scope.go:117] "RemoveContainer" containerID="d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"
Sep 16 13:57:03.348213 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:57:03.348194    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0\": container with ID starting with d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0 not found: ID does not exist" containerID="d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"
Sep 16 13:57:03.348273 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.348221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0"} err="failed to get container status \"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0\": rpc error: code = NotFound desc = could not find container \"d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0\": container with ID starting with d79364bef3e428f7b67f08fbb65a080ef5f7882059f4469d805d998d1b63a5b0 not found: ID does not exist"
Sep 16 13:57:03.348273 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.348238    2579 scope.go:117] "RemoveContainer" containerID="929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"
Sep 16 13:57:03.348514 ip-10-0-137-128 kubenswrapper[2579]: E0916 13:57:03.348486    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6\": container with ID starting with 929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6 not found: ID does not exist" containerID="929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"
Sep 16 13:57:03.348588 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.348528    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6"} err="failed to get container status \"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6\": rpc error: code = NotFound desc = could not find container \"929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6\": container with ID starting with 929b005316582ee76d1d86755c4145b54fd225d2d5a22411ae757f11bb887cb6 not found: ID does not exist"
Sep 16 13:57:03.355262 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.355236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "629d8529-625b-4889-8c4d-4b13222cc25c" (UID: "629d8529-625b-4889-8c4d-4b13222cc25c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:03.431176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431132    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431170    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/629d8529-625b-4889-8c4d-4b13222cc25c-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431176 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431181    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbk2\" (UniqueName: \"kubernetes.io/projected/629d8529-625b-4889-8c4d-4b13222cc25c-kube-api-access-klbk2\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431190    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629d8529-625b-4889-8c4d-4b13222cc25c-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431199    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431207    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.431457 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.431215    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/629d8529-625b-4889-8c4d-4b13222cc25c-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 13:57:03.646646 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.646607    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:57:03.651317 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:03.651288    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6664897865-jvmxj"]
Sep 16 13:57:04.457971 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:04.457940    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" path="/var/lib/kubelet/pods/629d8529-625b-4889-8c4d-4b13222cc25c/volumes"
Sep 16 13:57:27.828675 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828908    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="storage-initializer"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828919    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="storage-initializer"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828929    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828934    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828946    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.828952    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.829001    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="629d8529-625b-4889-8c4d-4b13222cc25c" containerName="main"
Sep 16 13:57:27.829075 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.829011    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="083a0ef3-5758-4b40-a8b0-661848efb60d" containerName="main"
Sep 16 13:57:27.833422 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.833396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.840760 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.840735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-lvhhc\""
Sep 16 13:57:27.840978 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.840958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:57:27.841098 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.841055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 13:57:27.841360 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.841322    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-8hhnz\""
Sep 16 13:57:27.841360 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.841335    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 13:57:27.841535 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.841335    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 13:57:27.841535 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.841345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 13:57:27.846850 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.846825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 13:57:27.919794 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.919752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btbp\" (UniqueName: \"kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.919794 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.919795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.919812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.919943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.919982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920040 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.920013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920430 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.920052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:27.920430 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:27.920083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021154 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btbp\" (UniqueName: \"kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021154 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021406 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021659 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021714 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021775 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.021908 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.021888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.022042 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.022023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.023705 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.023678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.023899 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.023881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.029932 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.029911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btbp\" (UniqueName: \"kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp\") pod \"custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.096073 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.095995    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 13:57:28.103411 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.103381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.106000 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.105971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-6v4gv\""
Sep 16 13:57:28.109701 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.109674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 13:57:28.121938 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.121910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpb2\" (UniqueName: \"kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.122051 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.121962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.143168 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.143135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:28.223391 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.223349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpb2\" (UniqueName: \"kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.223596 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.223443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.226002 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.225977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.231961 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.231934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpb2\" (UniqueName: \"kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.415679 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.415644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:28.420920 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.420888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 13:57:28.425256 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:57:28.425227    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36bbc50_df60_44be_a14e_c53929172b1f.slice/crio-feeaf4f52ae2b3ed6f2ce0f1bd2d821cddca604884a87f107c12bf029de496e9 WatchSource:0}: Error finding container feeaf4f52ae2b3ed6f2ce0f1bd2d821cddca604884a87f107c12bf029de496e9: Status 404 returned error can't find the container with id feeaf4f52ae2b3ed6f2ce0f1bd2d821cddca604884a87f107c12bf029de496e9
Sep 16 13:57:28.699375 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:28.699253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 13:57:28.701834 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:57:28.701799    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb539ba0_8ba6_4cf0_9c25_49d80de1497a.slice/crio-f220f1ea890cb1bead93afb4621b7cc97fca7c30e5c7a808a23397eb7c933b27 WatchSource:0}: Error finding container f220f1ea890cb1bead93afb4621b7cc97fca7c30e5c7a808a23397eb7c933b27: Status 404 returned error can't find the container with id f220f1ea890cb1bead93afb4621b7cc97fca7c30e5c7a808a23397eb7c933b27
Sep 16 13:57:29.424902 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.424864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerStarted","Data":"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"}
Sep 16 13:57:29.425417 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.424920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerStarted","Data":"feeaf4f52ae2b3ed6f2ce0f1bd2d821cddca604884a87f107c12bf029de496e9"}
Sep 16 13:57:29.425417 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.424950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:29.426688 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.426663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" event={"ID":"cb539ba0-8ba6-4cf0-9c25-49d80de1497a","Type":"ContainerStarted","Data":"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"}
Sep 16 13:57:29.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.426714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:57:29.426791 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.426738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" event={"ID":"cb539ba0-8ba6-4cf0-9c25-49d80de1497a","Type":"ContainerStarted","Data":"f220f1ea890cb1bead93afb4621b7cc97fca7c30e5c7a808a23397eb7c933b27"}
Sep 16 13:57:29.477858 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:29.477699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podStartSLOduration=1.477681718 podStartE2EDuration="1.477681718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:29.477083005 +0000 UTC m=+4155.515882128" watchObservedRunningTime="2026-09-16 13:57:29.477681718 +0000 UTC m=+4155.516480892"
Sep 16 13:57:30.432605 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:30.432543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerStarted","Data":"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"}
Sep 16 13:57:32.441666 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:32.441630    2579 generic.go:356] "Generic (PLEG): container finished" podID="f36bbc50-df60-44be-a14e-c53929172b1f" containerID="e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8" exitCode=0
Sep 16 13:57:32.442096 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:32.441697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerDied","Data":"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"}
Sep 16 13:57:33.448465 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:33.448427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerStarted","Data":"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"}
Sep 16 13:57:33.476053 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:33.475998    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podStartSLOduration=4.194940728 podStartE2EDuration="6.475978045s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.281037317s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:33.473551377 +0000 UTC m=+4159.512350521" watchObservedRunningTime="2026-09-16 13:57:33.475978045 +0000 UTC m=+4159.514777168"
Sep 16 13:57:38.143539 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:38.143492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:38.144001 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:38.143583    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:38.145024 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:38.144985    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:57:38.156556 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:38.156532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:57:40.778723 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.778687    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 13:57:40.782856 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.782830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.786928 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.786891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-fq9l8\""
Sep 16 13:57:40.787046 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.786968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-554tl\""
Sep 16 13:57:40.787124 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.787076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 13:57:40.787223 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.787198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 13:57:40.787391 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.787373    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 13:57:40.792636 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.792106    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 13:57:40.840028 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.839990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.840204 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.840067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dgj\" (UniqueName: \"kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.940962 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.940913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dgj\" (UniqueName: \"kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.941172 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.940981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.943742 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.943705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:40.950274 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:40.950239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dgj\" (UniqueName: \"kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:41.097107 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:41.097020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:41.382516 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:41.382423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 13:57:41.394978 ip-10-0-137-128 kubenswrapper[2579]: W0916 13:57:41.394938    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573b7f2b_690f_4523_a840_da384f5a5206.slice/crio-733c0c4058eb64739b9bd325c202c0538a1178ccbba86d4d2969fd3598c299ec WatchSource:0}: Error finding container 733c0c4058eb64739b9bd325c202c0538a1178ccbba86d4d2969fd3598c299ec: Status 404 returned error can't find the container with id 733c0c4058eb64739b9bd325c202c0538a1178ccbba86d4d2969fd3598c299ec
Sep 16 13:57:41.486184 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:41.486141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" event={"ID":"573b7f2b-690f-4523-a840-da384f5a5206","Type":"ContainerStarted","Data":"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"}
Sep 16 13:57:41.486434 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:41.486210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" event={"ID":"573b7f2b-690f-4523-a840-da384f5a5206","Type":"ContainerStarted","Data":"733c0c4058eb64739b9bd325c202c0538a1178ccbba86d4d2969fd3598c299ec"}
Sep 16 13:57:41.486996 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:41.486968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:57:48.143890 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:48.143832    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:57:58.144174 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:57:58.144117    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:00.434335 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:00.434300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 13:58:00.461059 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:00.461003    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podStartSLOduration=20.460987639 podStartE2EDuration="20.460987639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:41.518389976 +0000 UTC m=+4167.557189103" watchObservedRunningTime="2026-09-16 13:58:00.460987639 +0000 UTC m=+4186.499786762"
Sep 16 13:58:08.143900 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:08.143854    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:13.496482 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:13.496450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 13:58:14.684450 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:14.684419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:58:14.690657 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:14.690634    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 13:58:18.143690 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:18.143648    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:28.144432 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:28.144379    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:38.144494 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:38.144383    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:48.144543 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:48.144485    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:58:58.144003 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:58:58.143944    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:59:08.144119 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:08.144053    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:59:18.143746 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:18.143693    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8001/health\": dial tcp 10.134.0.81:8001: connect: connection refused"
Sep 16 13:59:28.154293 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:28.154254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:59:28.171021 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:28.170989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 13:59:59.878081 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:59.878036    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 13:59:59.883122 ip-10-0-137-128 kubenswrapper[2579]: I0916 13:59:59.883077    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 14:00:00.434684 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:00.434640    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:10.434667 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:10.434580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:14.903074 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:14.903028    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main" containerID="cri-o://cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd" gracePeriod=45
Sep 16 14:00:14.906378 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:14.906342    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" containerID="cri-o://9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573" gracePeriod=45
Sep 16 14:00:15.336315 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.336187    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 14:00:15.544419 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.544384    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="llm-d-routing-sidecar" containerID="cri-o://365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503" gracePeriod=45
Sep 16 14:00:15.787669 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.787642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 14:00:15.864238 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864238 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6btbp\" (UniqueName: \"kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864502 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs\") pod \"f36bbc50-df60-44be-a14e-c53929172b1f\" (UID: \"f36bbc50-df60-44be-a14e-c53929172b1f\") "
Sep 16 14:00:15.864848 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:00:15.864848 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.864794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.865074 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.865043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home" (OuterVolumeSpecName: "home") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.866952 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.866923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp" (OuterVolumeSpecName: "kube-api-access-6btbp") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "kube-api-access-6btbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:15.867230 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.867199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm" (OuterVolumeSpecName: "dshm") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.867321 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.867243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:00:15.876648 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.876618    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.888528 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.888504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f36bbc50-df60-44be-a14e-c53929172b1f" (UID: "f36bbc50-df60-44be-a14e-c53929172b1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965479    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36bbc50-df60-44be-a14e-c53929172b1f-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965510    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965519    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6btbp\" (UniqueName: \"kubernetes.io/projected/f36bbc50-df60-44be-a14e-c53929172b1f-kube-api-access-6btbp\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965529    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965537    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965545    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965553    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36bbc50-df60-44be-a14e-c53929172b1f-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.965592 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:15.965582    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36bbc50-df60-44be-a14e-c53929172b1f-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:16.189229 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189197    2579 generic.go:356] "Generic (PLEG): container finished" podID="f36bbc50-df60-44be-a14e-c53929172b1f" containerID="cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd" exitCode=0
Sep 16 14:00:16.189229 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189221    2579 generic.go:356] "Generic (PLEG): container finished" podID="f36bbc50-df60-44be-a14e-c53929172b1f" containerID="365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503" exitCode=0
Sep 16 14:00:16.189429 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"
Sep 16 14:00:16.189429 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerDied","Data":"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"}
Sep 16 14:00:16.189429 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerDied","Data":"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"}
Sep 16 14:00:16.189533 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f" event={"ID":"f36bbc50-df60-44be-a14e-c53929172b1f","Type":"ContainerDied","Data":"feeaf4f52ae2b3ed6f2ce0f1bd2d821cddca604884a87f107c12bf029de496e9"}
Sep 16 14:00:16.189533 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.189475    2579 scope.go:117] "RemoveContainer" containerID="cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"
Sep 16 14:00:16.198783 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.198759    2579 scope.go:117] "RemoveContainer" containerID="e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"
Sep 16 14:00:16.207060 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.207039    2579 scope.go:117] "RemoveContainer" containerID="365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"
Sep 16 14:00:16.213993 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.213965    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 14:00:16.215209 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.215194    2579 scope.go:117] "RemoveContainer" containerID="cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"
Sep 16 14:00:16.215513 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:00:16.215492    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd\": container with ID starting with cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd not found: ID does not exist" containerID="cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"
Sep 16 14:00:16.215716 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.215522    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"} err="failed to get container status \"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd\": rpc error: code = NotFound desc = could not find container \"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd\": container with ID starting with cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd not found: ID does not exist"
Sep 16 14:00:16.215716 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.215541    2579 scope.go:117] "RemoveContainer" containerID="e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"
Sep 16 14:00:16.215860 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:00:16.215839    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8\": container with ID starting with e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8 not found: ID does not exist" containerID="e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"
Sep 16 14:00:16.215903 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.215869    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"} err="failed to get container status \"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8\": rpc error: code = NotFound desc = could not find container \"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8\": container with ID starting with e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8 not found: ID does not exist"
Sep 16 14:00:16.215903 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.215887    2579 scope.go:117] "RemoveContainer" containerID="365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"
Sep 16 14:00:16.216232 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:00:16.216205    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503\": container with ID starting with 365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503 not found: ID does not exist" containerID="365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"
Sep 16 14:00:16.216334 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216247    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"} err="failed to get container status \"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503\": rpc error: code = NotFound desc = could not find container \"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503\": container with ID starting with 365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503 not found: ID does not exist"
Sep 16 14:00:16.216334 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216271    2579 scope.go:117] "RemoveContainer" containerID="cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"
Sep 16 14:00:16.216704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216599    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd"} err="failed to get container status \"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd\": rpc error: code = NotFound desc = could not find container \"cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd\": container with ID starting with cf8123e63b9575eddd4295107c9514af8f7e97b2ec013c8caebf91bbac36dbbd not found: ID does not exist"
Sep 16 14:00:16.216704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216632    2579 scope.go:117] "RemoveContainer" containerID="e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"
Sep 16 14:00:16.217033 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216928    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8"} err="failed to get container status \"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8\": rpc error: code = NotFound desc = could not find container \"e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8\": container with ID starting with e5fa7731d757ea31785795616f681ebe1d9da1a0ccdeba6bcaccecf8a3cbe9d8 not found: ID does not exist"
Sep 16 14:00:16.217033 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.216955    2579 scope.go:117] "RemoveContainer" containerID="365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"
Sep 16 14:00:16.217200 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.217177    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503"} err="failed to get container status \"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503\": rpc error: code = NotFound desc = could not find container \"365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503\": container with ID starting with 365fca22fbea2ff0d7ca04cbd4dcffee52132c87cf12879e294abd4c56d88503 not found: ID does not exist"
Sep 16 14:00:16.218474 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.218450    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-79cb8c997-rcf2f"]
Sep 16 14:00:16.458507 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:16.458460    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" path="/var/lib/kubelet/pods/f36bbc50-df60-44be-a14e-c53929172b1f/volumes"
Sep 16 14:00:20.434617 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:20.434577    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:20.435011 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:20.434682    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 14:00:23.496026 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:23.495973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:30.361378 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:30.361337    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main" containerID="cri-o://9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d" gracePeriod=45
Sep 16 14:00:30.434297 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:30.434255    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:33.495732 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:33.495688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:40.435133 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:40.435087    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:43.495771 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:43.495727    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:00:43.496226 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:43.495871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 14:00:45.145205 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.145176    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 14:00:45.215164 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.215065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs\") pod \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") "
Sep 16 14:00:45.215164 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.215142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlpb2\" (UniqueName: \"kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2\") pod \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\" (UID: \"cb539ba0-8ba6-4cf0-9c25-49d80de1497a\") "
Sep 16 14:00:45.217281 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.217244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb539ba0-8ba6-4cf0-9c25-49d80de1497a" (UID: "cb539ba0-8ba6-4cf0-9c25-49d80de1497a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:00:45.217407 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.217303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2" (OuterVolumeSpecName: "kube-api-access-vlpb2") pod "cb539ba0-8ba6-4cf0-9c25-49d80de1497a" (UID: "cb539ba0-8ba6-4cf0-9c25-49d80de1497a"). InnerVolumeSpecName "kube-api-access-vlpb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:45.312103 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.312063    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerID="9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573" exitCode=0
Sep 16 14:00:45.312274 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.312114    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"
Sep 16 14:00:45.312274 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.312143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" event={"ID":"cb539ba0-8ba6-4cf0-9c25-49d80de1497a","Type":"ContainerDied","Data":"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"}
Sep 16 14:00:45.312274 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.312201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9" event={"ID":"cb539ba0-8ba6-4cf0-9c25-49d80de1497a","Type":"ContainerDied","Data":"f220f1ea890cb1bead93afb4621b7cc97fca7c30e5c7a808a23397eb7c933b27"}
Sep 16 14:00:45.312274 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.312223    2579 scope.go:117] "RemoveContainer" containerID="9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"
Sep 16 14:00:45.315802 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.315782    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:45.315888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.315804    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpb2\" (UniqueName: \"kubernetes.io/projected/cb539ba0-8ba6-4cf0-9c25-49d80de1497a-kube-api-access-vlpb2\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:00:45.320979 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.320955    2579 scope.go:117] "RemoveContainer" containerID="9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"
Sep 16 14:00:45.321288 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:00:45.321266    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573\": container with ID starting with 9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573 not found: ID does not exist" containerID="9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"
Sep 16 14:00:45.321361 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.321301    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573"} err="failed to get container status \"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573\": rpc error: code = NotFound desc = could not find container \"9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573\": container with ID starting with 9405bf9a8c68a6416b418a46527e4f79abe5acfd2aadf9ed505c121dccd30573 not found: ID does not exist"
Sep 16 14:00:45.333756 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.333731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 14:00:45.337480 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:45.337457    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8fz9"]
Sep 16 14:00:46.457279 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:46.457241    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" path="/var/lib/kubelet/pods/cb539ba0-8ba6-4cf0-9c25-49d80de1497a/volumes"
Sep 16 14:00:53.496373 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:00:53.496328    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:01:00.611292 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.611258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 14:01:00.747473 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.747431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs\") pod \"573b7f2b-690f-4523-a840-da384f5a5206\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") "
Sep 16 14:01:00.747692 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.747507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6dgj\" (UniqueName: \"kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj\") pod \"573b7f2b-690f-4523-a840-da384f5a5206\" (UID: \"573b7f2b-690f-4523-a840-da384f5a5206\") "
Sep 16 14:01:00.749859 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.749834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj" (OuterVolumeSpecName: "kube-api-access-m6dgj") pod "573b7f2b-690f-4523-a840-da384f5a5206" (UID: "573b7f2b-690f-4523-a840-da384f5a5206"). InnerVolumeSpecName "kube-api-access-m6dgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:00.749930 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.749832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "573b7f2b-690f-4523-a840-da384f5a5206" (UID: "573b7f2b-690f-4523-a840-da384f5a5206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:01:00.848228 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.848185    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6dgj\" (UniqueName: \"kubernetes.io/projected/573b7f2b-690f-4523-a840-da384f5a5206-kube-api-access-m6dgj\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:01:00.848228 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:00.848222    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573b7f2b-690f-4523-a840-da384f5a5206-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:01:01.385713 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.385668    2579 generic.go:356] "Generic (PLEG): container finished" podID="573b7f2b-690f-4523-a840-da384f5a5206" containerID="9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d" exitCode=0
Sep 16 14:01:01.385932 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.385737    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"
Sep 16 14:01:01.385932 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.385762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" event={"ID":"573b7f2b-690f-4523-a840-da384f5a5206","Type":"ContainerDied","Data":"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"}
Sep 16 14:01:01.385932 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.385855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj" event={"ID":"573b7f2b-690f-4523-a840-da384f5a5206","Type":"ContainerDied","Data":"733c0c4058eb64739b9bd325c202c0538a1178ccbba86d4d2969fd3598c299ec"}
Sep 16 14:01:01.385932 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.385888    2579 scope.go:117] "RemoveContainer" containerID="9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"
Sep 16 14:01:01.395068 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.395048    2579 scope.go:117] "RemoveContainer" containerID="9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"
Sep 16 14:01:01.395341 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:01:01.395320    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d\": container with ID starting with 9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d not found: ID does not exist" containerID="9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"
Sep 16 14:01:01.395394 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.395348    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d"} err="failed to get container status \"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d\": rpc error: code = NotFound desc = could not find container \"9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d\": container with ID starting with 9b5ab0fe00f646785bacdf4f0563a4e86fd046135b67fc55ca217d2b0aaa068d not found: ID does not exist"
Sep 16 14:01:01.414127 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.414087    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 14:01:01.419413 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:01.419381    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-ptzpj"]
Sep 16 14:01:02.458016 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:02.457976    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573b7f2b-690f-4523-a840-da384f5a5206" path="/var/lib/kubelet/pods/573b7f2b-690f-4523-a840-da384f5a5206/volumes"
Sep 16 14:01:04.263497 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263452    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263871    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263892    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263906    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263914    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="storage-initializer"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263938    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="storage-initializer"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263954    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="llm-d-routing-sidecar"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="llm-d-routing-sidecar"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263985    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main"
Sep 16 14:01:04.264005 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.263995    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main"
Sep 16 14:01:04.264506 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.264084    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb539ba0-8ba6-4cf0-9c25-49d80de1497a" containerName="main"
Sep 16 14:01:04.264506 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.264097    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="main"
Sep 16 14:01:04.264506 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.264107    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="573b7f2b-690f-4523-a840-da384f5a5206" containerName="main"
Sep 16 14:01:04.264506 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.264116    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36bbc50-df60-44be-a14e-c53929172b1f" containerName="llm-d-routing-sidecar"
Sep 16 14:01:04.268983 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.268956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.271913 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.271881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 14:01:04.272084 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.272027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 14:01:04.272084 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.272063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 14:01:04.272540 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.272520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rphcc\""
Sep 16 14:01:04.272668 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.272589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 14:01:04.273310 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.273291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 14:01:04.273404 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.273296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-5x9xb\""
Sep 16 14:01:04.284171 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.284138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:01:04.378701 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378701 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.378985 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.378945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c48w\" (UniqueName: \"kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479382 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479382 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.479704 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c48w\" (UniqueName: \"kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.480025 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.480025 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.480025 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.479954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.480135 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.480103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.480365 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.480347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.482456 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.482434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.482654 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.482635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.488933 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.488912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c48w\" (UniqueName: \"kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w\") pod \"router-with-refs-pd-test-kserve-854c4d7c49-fjvck\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.579526 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.579422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:04.680715 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.680678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:01:04.684121 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.684102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.687229 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.687207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-hsqxq\""
Sep 16 14:01:04.693825 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.693798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:01:04.782159 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.782114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.782342 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.782217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79p9\" (UniqueName: \"kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.865113 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.865027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:01:04.868724 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:01:04.868695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0704042a_2f10_4611_a862_cf4b6e5cebeb.slice/crio-fffdc48371975149695ea97a2d68795e0727864abdab9df10058a80bb47772fd WatchSource:0}: Error finding container fffdc48371975149695ea97a2d68795e0727864abdab9df10058a80bb47772fd: Status 404 returned error can't find the container with id fffdc48371975149695ea97a2d68795e0727864abdab9df10058a80bb47772fd
Sep 16 14:01:04.883305 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.883283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.883413 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.883397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79p9\" (UniqueName: \"kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.886325 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.886292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.891894 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.891873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79p9\" (UniqueName: \"kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:04.995690 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:04.995648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:05.269431 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.269403    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:01:05.271847 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:01:05.271819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4faaf31_80bf_484a_a036_220957d10e9c.slice/crio-c1a0955eddf9e0c0bca47bb3e9e9d49dfd54d3577ace08d7f7ab29ae5af4b54a WatchSource:0}: Error finding container c1a0955eddf9e0c0bca47bb3e9e9d49dfd54d3577ace08d7f7ab29ae5af4b54a: Status 404 returned error can't find the container with id c1a0955eddf9e0c0bca47bb3e9e9d49dfd54d3577ace08d7f7ab29ae5af4b54a
Sep 16 14:01:05.414527 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.414487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" event={"ID":"a4faaf31-80bf-484a-a036-220957d10e9c","Type":"ContainerStarted","Data":"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"}
Sep 16 14:01:05.414737 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.414591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:05.414737 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.414622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" event={"ID":"a4faaf31-80bf-484a-a036-220957d10e9c","Type":"ContainerStarted","Data":"c1a0955eddf9e0c0bca47bb3e9e9d49dfd54d3577ace08d7f7ab29ae5af4b54a"}
Sep 16 14:01:05.415949 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.415920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerStarted","Data":"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"}
Sep 16 14:01:05.416088 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.415967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerStarted","Data":"fffdc48371975149695ea97a2d68795e0727864abdab9df10058a80bb47772fd"}
Sep 16 14:01:05.416088 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.416008    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:05.436697 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:05.436640    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podStartSLOduration=1.436620105 podStartE2EDuration="1.436620105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:05.435010601 +0000 UTC m=+4371.473809724" watchObservedRunningTime="2026-09-16 14:01:05.436620105 +0000 UTC m=+4371.475419225"
Sep 16 14:01:06.421452 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:06.421409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerStarted","Data":"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"}
Sep 16 14:01:08.431806 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:08.431765    2579 generic.go:356] "Generic (PLEG): container finished" podID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerID="69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f" exitCode=0
Sep 16 14:01:08.432214 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:08.431844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerDied","Data":"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"}
Sep 16 14:01:09.438907 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:09.438874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerStarted","Data":"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"}
Sep 16 14:01:09.469425 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:09.469364    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podStartSLOduration=3.102796278 podStartE2EDuration="5.469341997s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.366545719s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:09.467805998 +0000 UTC m=+4375.506605133" watchObservedRunningTime="2026-09-16 14:01:09.469341997 +0000 UTC m=+4375.508141120"
Sep 16 14:01:11.750417 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.750383    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:01:11.752917 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.752885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.757171 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.757141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nsrcv\""
Sep 16 14:01:11.757304 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.757141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-4q2n7\""
Sep 16 14:01:11.757304 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.757278    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 14:01:11.757432 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.757416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 14:01:11.757506 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.757490    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 14:01:11.770737 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.770707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:01:11.840896 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.840856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.841099 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.840954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcnt4\" (UniqueName: \"kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.942030 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.941982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.942248 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.942054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcnt4\" (UniqueName: \"kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.944721 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.944692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:11.953938 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:11.953910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcnt4\" (UniqueName: \"kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:12.063602 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.063462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:12.343508 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.343366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:01:12.346619 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:01:12.346589    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaefa5b1_5f6b_472a_ac64_32ed7c1d898b.slice/crio-62ceebfd92445512ef14d124a366c0b2278e249f26d002e2376aa57e64cde45f WatchSource:0}: Error finding container 62ceebfd92445512ef14d124a366c0b2278e249f26d002e2376aa57e64cde45f: Status 404 returned error can't find the container with id 62ceebfd92445512ef14d124a366c0b2278e249f26d002e2376aa57e64cde45f
Sep 16 14:01:12.458406 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.458372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:12.458600 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.458425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" event={"ID":"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b","Type":"ContainerStarted","Data":"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"}
Sep 16 14:01:12.458600 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.458450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" event={"ID":"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b","Type":"ContainerStarted","Data":"62ceebfd92445512ef14d124a366c0b2278e249f26d002e2376aa57e64cde45f"}
Sep 16 14:01:12.477748 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:12.477691    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podStartSLOduration=1.477649215 podStartE2EDuration="1.477649215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:12.476916955 +0000 UTC m=+4378.515716075" watchObservedRunningTime="2026-09-16 14:01:12.477649215 +0000 UTC m=+4378.516448315"
Sep 16 14:01:14.580196 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:14.580156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:14.582195 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:14.582165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:14.582399 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:14.582364    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:01:14.600189 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:14.600152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:01:24.580705 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:24.580657    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:01:34.580813 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:34.580758    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:01:36.423241 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:36.423208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:01:43.463333 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:43.463299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:01:44.580039 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:44.579990    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:01:54.580789 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:01:54.580737    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:04.580786 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:04.580739    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:14.580212 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:14.580159    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:24.580210 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:24.580154    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:34.580810 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:34.580750    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:44.580917 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:44.580862    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:02:54.580234 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:02:54.580187    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 14:03:04.590255 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:03:04.590218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:03:04.601843 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:03:04.601818    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:03:14.713459 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:03:14.713417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 14:03:14.719909 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:03:14.719885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 14:04:04.537197 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:04.537150    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:04:04.545224 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:04.545191    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:04:05.923419 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:05.923378    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:04:06.423732 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:06.423693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:13.462994 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:13.462953    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:16.423949 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:16.423909    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:19.557289 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:19.557242    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" containerID="cri-o://07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b" gracePeriod=45
Sep 16 14:04:19.567483 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:19.567446    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main" containerID="cri-o://7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7" gracePeriod=45
Sep 16 14:04:20.183793 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.183757    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="llm-d-routing-sidecar" containerID="cri-o://ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553" gracePeriod=45
Sep 16 14:04:20.297293 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.297256    2579 generic.go:356] "Generic (PLEG): container finished" podID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerID="7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7" exitCode=0
Sep 16 14:04:20.297449 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.297323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerDied","Data":"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"}
Sep 16 14:04:20.427523 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.427495    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.454985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c48w\" (UniqueName: \"kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w\") pod \"0704042a-2f10-4611-a862-cf4b6e5cebeb\" (UID: \"0704042a-2f10-4611-a862-cf4b6e5cebeb\") "
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.455075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.455888 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.455275    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.456531 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.455982    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home" (OuterVolumeSpecName: "home") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.456531 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.456126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:04:20.458733 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.458703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:04:20.459093 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.459068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w" (OuterVolumeSpecName: "kube-api-access-2c48w") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "kube-api-access-2c48w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:20.459353 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.459320    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm" (OuterVolumeSpecName: "dshm") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.469134 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.469106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.485245 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.485213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0704042a-2f10-4611-a862-cf4b6e5cebeb" (UID: "0704042a-2f10-4611-a862-cf4b6e5cebeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.556359 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556316    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0704042a-2f10-4611-a862-cf4b6e5cebeb-cabundle-cert\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556359 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556351    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0704042a-2f10-4611-a862-cf4b6e5cebeb-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556359 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556363    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-kserve-provision-location\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556623 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556371    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556623 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556382    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556623 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556390    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c48w\" (UniqueName: \"kubernetes.io/projected/0704042a-2f10-4611-a862-cf4b6e5cebeb-kube-api-access-2c48w\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.556623 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.556402    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0704042a-2f10-4611-a862-cf4b6e5cebeb-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.942745 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:20.942700    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main" containerID="cri-o://f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1" gracePeriod=45
Sep 16 14:04:21.303329 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.303235    2579 generic.go:356] "Generic (PLEG): container finished" podID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerID="ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553" exitCode=0
Sep 16 14:04:21.303329 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.303287    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"
Sep 16 14:04:21.303329 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.303316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerDied","Data":"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"}
Sep 16 14:04:21.303626 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.303379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck" event={"ID":"0704042a-2f10-4611-a862-cf4b6e5cebeb","Type":"ContainerDied","Data":"fffdc48371975149695ea97a2d68795e0727864abdab9df10058a80bb47772fd"}
Sep 16 14:04:21.303626 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.303412    2579 scope.go:117] "RemoveContainer" containerID="7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"
Sep 16 14:04:21.312344 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.312323    2579 scope.go:117] "RemoveContainer" containerID="69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"
Sep 16 14:04:21.320236 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.320217    2579 scope.go:117] "RemoveContainer" containerID="ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"
Sep 16 14:04:21.329934 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.329912    2579 scope.go:117] "RemoveContainer" containerID="7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"
Sep 16 14:04:21.330244 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:04:21.330220    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7\": container with ID starting with 7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7 not found: ID does not exist" containerID="7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"
Sep 16 14:04:21.330355 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.330284    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7"} err="failed to get container status \"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7\": rpc error: code = NotFound desc = could not find container \"7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7\": container with ID starting with 7a8452db5898dbb9364c66dc21b071dd8f677c6c9695c28b69c9eb095ef752f7 not found: ID does not exist"
Sep 16 14:04:21.330355 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.330310    2579 scope.go:117] "RemoveContainer" containerID="69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"
Sep 16 14:04:21.330650 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:04:21.330625    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f\": container with ID starting with 69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f not found: ID does not exist" containerID="69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"
Sep 16 14:04:21.330650 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.330658    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f"} err="failed to get container status \"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f\": rpc error: code = NotFound desc = could not find container \"69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f\": container with ID starting with 69027531020c14f9d56547a2c2d5a91c2233695589dd6e67de5461981d64141f not found: ID does not exist"
Sep 16 14:04:21.330650 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.330675    2579 scope.go:117] "RemoveContainer" containerID="ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"
Sep 16 14:04:21.330965 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:04:21.330948    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553\": container with ID starting with ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553 not found: ID does not exist" containerID="ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"
Sep 16 14:04:21.331020 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.330969    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553"} err="failed to get container status \"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553\": rpc error: code = NotFound desc = could not find container \"ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553\": container with ID starting with ad7b14c6b531b8086f56f7e11b7991ddb160c655e9c80fb8abcec505fe92d553 not found: ID does not exist"
Sep 16 14:04:21.331357 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.331335    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:04:21.336417 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:21.336394    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-854c4d7c49-fjvck"]
Sep 16 14:04:22.458061 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:22.458027    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" path="/var/lib/kubelet/pods/0704042a-2f10-4611-a862-cf4b6e5cebeb/volumes"
Sep 16 14:04:23.463267 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:23.463221    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:26.423666 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:26.423629    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:26.424076 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:26.423738    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:04:33.463369 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:33.463327    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:33.463794 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:33.463438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:04:36.423029 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:36.422989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:43.463146 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:43.463104    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:46.422859 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:46.422819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:04:49.805793 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.805765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:04:49.884907 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.884807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs\") pod \"a4faaf31-80bf-484a-a036-220957d10e9c\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") "
Sep 16 14:04:49.884907 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.884883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l79p9\" (UniqueName: \"kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9\") pod \"a4faaf31-80bf-484a-a036-220957d10e9c\" (UID: \"a4faaf31-80bf-484a-a036-220957d10e9c\") "
Sep 16 14:04:49.886972 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.886938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4faaf31-80bf-484a-a036-220957d10e9c" (UID: "a4faaf31-80bf-484a-a036-220957d10e9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:04:49.887101 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.887051    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9" (OuterVolumeSpecName: "kube-api-access-l79p9") pod "a4faaf31-80bf-484a-a036-220957d10e9c" (UID: "a4faaf31-80bf-484a-a036-220957d10e9c"). InnerVolumeSpecName "kube-api-access-l79p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:49.985734 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.985691    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4faaf31-80bf-484a-a036-220957d10e9c-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:49.985734 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:49.985726    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79p9\" (UniqueName: \"kubernetes.io/projected/a4faaf31-80bf-484a-a036-220957d10e9c-kube-api-access-l79p9\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:50.424945 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.424910    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4faaf31-80bf-484a-a036-220957d10e9c" containerID="07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b" exitCode=0
Sep 16 14:04:50.425140 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.424956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"
Sep 16 14:04:50.425140 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.424960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" event={"ID":"a4faaf31-80bf-484a-a036-220957d10e9c","Type":"ContainerDied","Data":"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"}
Sep 16 14:04:50.425140 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.425103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4" event={"ID":"a4faaf31-80bf-484a-a036-220957d10e9c","Type":"ContainerDied","Data":"c1a0955eddf9e0c0bca47bb3e9e9d49dfd54d3577ace08d7f7ab29ae5af4b54a"}
Sep 16 14:04:50.425324 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.425147    2579 scope.go:117] "RemoveContainer" containerID="07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"
Sep 16 14:04:50.433916 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.433897    2579 scope.go:117] "RemoveContainer" containerID="07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"
Sep 16 14:04:50.434183 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:04:50.434162    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b\": container with ID starting with 07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b not found: ID does not exist" containerID="07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"
Sep 16 14:04:50.434234 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.434192    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b"} err="failed to get container status \"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b\": rpc error: code = NotFound desc = could not find container \"07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b\": container with ID starting with 07c13e5ee0343dc15459860c3b305133474a073f6b0e4ce319e118e579d0988b not found: ID does not exist"
Sep 16 14:04:50.447106 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.447083    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:04:50.457172 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:50.457152    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45swrd4"]
Sep 16 14:04:51.185761 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.185734    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:04:51.297375 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.297278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcnt4\" (UniqueName: \"kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4\") pod \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") "
Sep 16 14:04:51.297375 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.297369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs\") pod \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\" (UID: \"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b\") "
Sep 16 14:04:51.299625 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.299596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4" (OuterVolumeSpecName: "kube-api-access-wcnt4") pod "aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" (UID: "aaefa5b1-5f6b-472a-ac64-32ed7c1d898b"). InnerVolumeSpecName "kube-api-access-wcnt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:51.299740 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.299667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" (UID: "aaefa5b1-5f6b-472a-ac64-32ed7c1d898b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:04:51.397983 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.397944    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:51.397983 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.397982    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcnt4\" (UniqueName: \"kubernetes.io/projected/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b-kube-api-access-wcnt4\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:04:51.430597 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.430537    2579 generic.go:356] "Generic (PLEG): container finished" podID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerID="f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1" exitCode=0
Sep 16 14:04:51.430597 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.430595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"
Sep 16 14:04:51.430816 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.430618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" event={"ID":"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b","Type":"ContainerDied","Data":"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"}
Sep 16 14:04:51.430816 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.430693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r" event={"ID":"aaefa5b1-5f6b-472a-ac64-32ed7c1d898b","Type":"ContainerDied","Data":"62ceebfd92445512ef14d124a366c0b2278e249f26d002e2376aa57e64cde45f"}
Sep 16 14:04:51.430816 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.430716    2579 scope.go:117] "RemoveContainer" containerID="f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"
Sep 16 14:04:51.451507 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.451485    2579 scope.go:117] "RemoveContainer" containerID="f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"
Sep 16 14:04:51.451838 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:04:51.451816    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1\": container with ID starting with f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1 not found: ID does not exist" containerID="f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"
Sep 16 14:04:51.451924 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.451847    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1"} err="failed to get container status \"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1\": rpc error: code = NotFound desc = could not find container \"f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1\": container with ID starting with f68e868f0a9445baf352427cb9b1e4e34be8fcbdc0bc7c608e6fcf34436732b1 not found: ID does not exist"
Sep 16 14:04:51.462189 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.462163    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:04:51.463732 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:51.463710    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-drg8r"]
Sep 16 14:04:52.457317 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:52.457278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" path="/var/lib/kubelet/pods/a4faaf31-80bf-484a-a036-220957d10e9c/volumes"
Sep 16 14:04:52.457723 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:04:52.457704    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" path="/var/lib/kubelet/pods/aaefa5b1-5f6b-472a-ac64-32ed7c1d898b/volumes"
Sep 16 14:05:03.798203 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798161    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798465    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798470    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798482    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="llm-d-routing-sidecar"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="llm-d-routing-sidecar"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798496    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="storage-initializer"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798501    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="storage-initializer"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798509    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798514    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798556    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="llm-d-routing-sidecar"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798581    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4faaf31-80bf-484a-a036-220957d10e9c" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798588    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0704042a-2f10-4611-a862-cf4b6e5cebeb" containerName="main"
Sep 16 14:05:03.798601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.798594    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaefa5b1-5f6b-472a-ac64-32ed7c1d898b" containerName="main"
Sep 16 14:05:03.802680 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.802658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.806513 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.806473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 14:05:03.806669 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.806473    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-zsktt\""
Sep 16 14:05:03.806669 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.806518    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 14:05:03.806669 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.806606    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 14:05:03.813453 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.813426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:03.907410 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.907410 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.907670 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvtl\" (UniqueName: \"kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.907670 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.907670 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:03.907670 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:03.907647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.008838 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.008796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.008838 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.008844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009054 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.008965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009100 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009100 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009182 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvtl\" (UniqueName: \"kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009234 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009321 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.009429 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.009407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.011210 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.011186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.011578 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.011545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.017839 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.017808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvtl\" (UniqueName: \"kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-mp78m\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.063240 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.063156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:05:04.066799 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.066776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.070169 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.070145    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-dmqpd\""
Sep 16 14:05:04.076084 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.076052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:05:04.112741 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.112708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:04.210158 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.210118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxxr\" (UniqueName: \"kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.210323 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.210189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.311030 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.310985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxxr\" (UniqueName: \"kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.311254 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.311052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.313657 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.313602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.321667 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.321644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxxr\" (UniqueName: \"kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.377603 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.377541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:04.388859 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.388830    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:04.391150 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:05:04.391121    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775f8627_557f_4ce2_84d5_f2ee20c4ff51.slice/crio-47e91d1ee71dc2326cce7dcc6682526f17c89b35ca9fc149d8bbf394a888026f WatchSource:0}: Error finding container 47e91d1ee71dc2326cce7dcc6682526f17c89b35ca9fc149d8bbf394a888026f: Status 404 returned error can't find the container with id 47e91d1ee71dc2326cce7dcc6682526f17c89b35ca9fc149d8bbf394a888026f
Sep 16 14:05:04.495465 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.495423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" event={"ID":"775f8627-557f-4ce2-84d5-f2ee20c4ff51","Type":"ContainerStarted","Data":"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"}
Sep 16 14:05:04.495688 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.495486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" event={"ID":"775f8627-557f-4ce2-84d5-f2ee20c4ff51","Type":"ContainerStarted","Data":"47e91d1ee71dc2326cce7dcc6682526f17c89b35ca9fc149d8bbf394a888026f"}
Sep 16 14:05:04.654438 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.654389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" podStartSLOduration=1.654370761 podStartE2EDuration="1.654370761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:05:04.517132826 +0000 UTC m=+4610.555931939" watchObservedRunningTime="2026-09-16 14:05:04.654370761 +0000 UTC m=+4610.693169885"
Sep 16 14:05:04.655239 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:04.655214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:05:04.656796 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:05:04.656757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f21a8a_d663_4266_8db1_067c356b9348.slice/crio-d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573 WatchSource:0}: Error finding container d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573: Status 404 returned error can't find the container with id d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573
Sep 16 14:05:05.113230 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:05.113141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:05.506277 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:05.506238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" event={"ID":"22f21a8a-d663-4266-8db1-067c356b9348","Type":"ContainerStarted","Data":"286b9fd6502bbbea268b90450559769fabf15ed807244e9fe45da2c4a70cb653"}
Sep 16 14:05:05.506493 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:05.506322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" event={"ID":"22f21a8a-d663-4266-8db1-067c356b9348","Type":"ContainerStarted","Data":"d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573"}
Sep 16 14:05:05.529414 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:05.529361    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podStartSLOduration=1.529344852 podStartE2EDuration="1.529344852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:05:05.528720337 +0000 UTC m=+4611.567519460" watchObservedRunningTime="2026-09-16 14:05:05.529344852 +0000 UTC m=+4611.568143975"
Sep 16 14:05:06.510946 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:06.510905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:14.113534 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:14.113493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:14.126279 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:14.126245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:14.557060 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:14.557028    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:37.517256 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:37.517225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:05:48.614605 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:48.614548    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:48.622891 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:48.622856    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:05:48.624254 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:05:48.624212    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 14:05:48.624254 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.624254 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.624254 ip-10-0-137-128 kubenswrapper[2579]:  > containerID="d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527" cmd=["/bin/sleep","15"]
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:48.624268    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" message="" err=<
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:  >
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:05:48.624281    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerName="main" containerID="cri-o://d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"
Sep 16 14:05:48.624580 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:48.624321    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerName="main" containerID="cri-o://d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527" gracePeriod=60
Sep 16 14:05:49.373847 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.373821    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:49.489868 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.489868 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.490110 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489896    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.490110 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.490110 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlvtl\" (UniqueName: \"kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.490110 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.489990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs\") pod \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\" (UID: \"775f8627-557f-4ce2-84d5-f2ee20c4ff51\") "
Sep 16 14:05:49.490287 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.490168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home" (OuterVolumeSpecName: "home") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:49.490287 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.490183    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:49.490287 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.490206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache" (OuterVolumeSpecName: "model-cache") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:49.492164 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.492138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl" (OuterVolumeSpecName: "kube-api-access-dlvtl") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "kube-api-access-dlvtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:49.492263 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.492171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm" (OuterVolumeSpecName: "dshm") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:49.492263 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.492221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "775f8627-557f-4ce2-84d5-f2ee20c4ff51" (UID: "775f8627-557f-4ce2-84d5-f2ee20c4ff51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:05:49.591469 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591427    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-home\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.591469 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591463    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tmp-dir\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.591469 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591478    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-model-cache\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.591740 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvtl\" (UniqueName: \"kubernetes.io/projected/775f8627-557f-4ce2-84d5-f2ee20c4ff51-kube-api-access-dlvtl\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.591740 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591501    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775f8627-557f-4ce2-84d5-f2ee20c4ff51-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.591740 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.591512    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775f8627-557f-4ce2-84d5-f2ee20c4ff51-dshm\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.697354 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.697316    2579 generic.go:356] "Generic (PLEG): container finished" podID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerID="d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527" exitCode=0
Sep 16 14:05:49.697863 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.697371    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"
Sep 16 14:05:49.697863 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.697399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" event={"ID":"775f8627-557f-4ce2-84d5-f2ee20c4ff51","Type":"ContainerDied","Data":"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"}
Sep 16 14:05:49.697863 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.697455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m" event={"ID":"775f8627-557f-4ce2-84d5-f2ee20c4ff51","Type":"ContainerDied","Data":"47e91d1ee71dc2326cce7dcc6682526f17c89b35ca9fc149d8bbf394a888026f"}
Sep 16 14:05:49.697863 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.697479    2579 scope.go:117] "RemoveContainer" containerID="d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"
Sep 16 14:05:49.706197 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.706179    2579 scope.go:117] "RemoveContainer" containerID="d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"
Sep 16 14:05:49.706483 ip-10-0-137-128 kubenswrapper[2579]: E0916 14:05:49.706464    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527\": container with ID starting with d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527 not found: ID does not exist" containerID="d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"
Sep 16 14:05:49.706536 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.706493    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527"} err="failed to get container status \"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527\": rpc error: code = NotFound desc = could not find container \"d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527\": container with ID starting with d367d61fdd9655fd0bdc3a05c53afc7699ff8a23bed9b0af09032eeaffa8d527 not found: ID does not exist"
Sep 16 14:05:49.719174 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.719149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:49.725226 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:49.725205    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-mp78m"]
Sep 16 14:05:50.457833 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:50.457791    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" path="/var/lib/kubelet/pods/775f8627-557f-4ce2-84d5-f2ee20c4ff51/volumes"
Sep 16 14:05:57.516891 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:05:57.516843    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:06:03.642057 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:03.642010    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main" containerID="cri-o://286b9fd6502bbbea268b90450559769fabf15ed807244e9fe45da2c4a70cb653" gracePeriod=45
Sep 16 14:06:07.517868 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:07.517782    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:06:17.517275 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:17.517232    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:06:17.517682 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:17.517348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:06:27.517124 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:27.517085    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 14:06:33.880458 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.880427    2579 generic.go:356] "Generic (PLEG): container finished" podID="22f21a8a-d663-4266-8db1-067c356b9348" containerID="286b9fd6502bbbea268b90450559769fabf15ed807244e9fe45da2c4a70cb653" exitCode=0
Sep 16 14:06:33.880869 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.880507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" event={"ID":"22f21a8a-d663-4266-8db1-067c356b9348","Type":"ContainerDied","Data":"286b9fd6502bbbea268b90450559769fabf15ed807244e9fe45da2c4a70cb653"}
Sep 16 14:06:33.880869 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.880592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2" event={"ID":"22f21a8a-d663-4266-8db1-067c356b9348","Type":"ContainerDied","Data":"d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573"}
Sep 16 14:06:33.880869 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.880620    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2543420f513005801128809b211167c94a716e52ae5d1fc2fd9f6f98bbef573"
Sep 16 14:06:33.887167 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.887147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:06:33.944625 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.944529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs\") pod \"22f21a8a-d663-4266-8db1-067c356b9348\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") "
Sep 16 14:06:33.944625 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.944589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzxxr\" (UniqueName: \"kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr\") pod \"22f21a8a-d663-4266-8db1-067c356b9348\" (UID: \"22f21a8a-d663-4266-8db1-067c356b9348\") "
Sep 16 14:06:33.946729 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.946702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr" (OuterVolumeSpecName: "kube-api-access-mzxxr") pod "22f21a8a-d663-4266-8db1-067c356b9348" (UID: "22f21a8a-d663-4266-8db1-067c356b9348"). InnerVolumeSpecName "kube-api-access-mzxxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:33.946729 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:33.946715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22f21a8a-d663-4266-8db1-067c356b9348" (UID: "22f21a8a-d663-4266-8db1-067c356b9348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:06:34.046080 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:34.046035    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f21a8a-d663-4266-8db1-067c356b9348-tls-certs\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:06:34.046080 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:34.046070    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxxr\" (UniqueName: \"kubernetes.io/projected/22f21a8a-d663-4266-8db1-067c356b9348-kube-api-access-mzxxr\") on node \"ip-10-0-137-128.ec2.internal\" DevicePath \"\""
Sep 16 14:06:34.884548 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:34.884466    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"
Sep 16 14:06:34.908260 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:34.908225    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:06:34.909998 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:34.909975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8zng2"]
Sep 16 14:06:36.458047 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:06:36.458008    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f21a8a-d663-4266-8db1-067c356b9348" path="/var/lib/kubelet/pods/22f21a8a-d663-4266-8db1-067c356b9348/volumes"
Sep 16 14:07:00.030800 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.030761    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/0.log"
Sep 16 14:07:00.043470 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.043444    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/1.log"
Sep 16 14:07:00.065483 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.065456    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 14:07:00.848820 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.848778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/0.log"
Sep 16 14:07:00.861524 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.861499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/1.log"
Sep 16 14:07:00.874473 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:00.874449    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 14:07:01.887195 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:01.887144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-mtr9c_4f1508c2-2d2a-4055-b0ad-20da138f0a78/authorino/0.log"
Sep 16 14:07:01.946745 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:01.946714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d2zkc_e5d00af6-0c0e-48c3-a01a-af1638ff891d/kuadrant-console-plugin/0.log"
Sep 16 14:07:12.699238 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:12.699201    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-74lv5_d4820864-53d4-4b54-82dd-996b2be796ec/global-pull-secret-syncer/0.log"
Sep 16 14:07:12.872439 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:12.872409    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rc64q_7b6e0fb6-82c9-4ed3-9734-16bb78615874/konnectivity-agent/0.log"
Sep 16 14:07:12.960977 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:12.960900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-128.ec2.internal_3bfe345cfe49aded3ae47813e63145e8/haproxy/0.log"
Sep 16 14:07:18.742645 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:18.742602    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-mtr9c_4f1508c2-2d2a-4055-b0ad-20da138f0a78/authorino/0.log"
Sep 16 14:07:18.845974 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:18.845941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d2zkc_e5d00af6-0c0e-48c3-a01a-af1638ff891d/kuadrant-console-plugin/0.log"
Sep 16 14:07:20.831854 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:20.831820    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-df7jd_52efc589-4291-480d-8014-961d6ec84d43/node-exporter/0.log"
Sep 16 14:07:20.859280 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:20.859249    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-df7jd_52efc589-4291-480d-8014-961d6ec84d43/kube-rbac-proxy/0.log"
Sep 16 14:07:20.889391 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:20.889367    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-df7jd_52efc589-4291-480d-8014-961d6ec84d43/init-textfile/0.log"
Sep 16 14:07:21.285762 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:21.285721    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-v5bmn_c9f2b254-98d3-42a4-a47c-780941d2b3d2/prometheus-operator/0.log"
Sep 16 14:07:21.311301 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:21.311268    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-v5bmn_c9f2b254-98d3-42a4-a47c-780941d2b3d2/kube-rbac-proxy/0.log"
Sep 16 14:07:22.449706 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.449672    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"]
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.449970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main"
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.449981    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main"
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.449990    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerName="main"
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.449996    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerName="main"
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.450046    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="775f8627-557f-4ce2-84d5-f2ee20c4ff51" containerName="main"
Sep 16 14:07:22.450091 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.450056    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f21a8a-d663-4266-8db1-067c356b9348" containerName="main"
Sep 16 14:07:22.453004 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.452978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.456695 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.456667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87z4h\"/\"default-dockercfg-4xkqf\""
Sep 16 14:07:22.457303 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.457285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87z4h\"/\"kube-root-ca.crt\""
Sep 16 14:07:22.458118 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.458099    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87z4h\"/\"openshift-service-ca.crt\""
Sep 16 14:07:22.468896 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.468873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"]
Sep 16 14:07:22.526410 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.526372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmt2\" (UniqueName: \"kubernetes.io/projected/448b1272-2c04-435d-b909-ad97a75c32b8-kube-api-access-nqmt2\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.526601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.526426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-podres\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.526601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.526484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-proc\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.526601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.526543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-sys\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.526601 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.526581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-lib-modules\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.626993 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.626960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmt2\" (UniqueName: \"kubernetes.io/projected/448b1272-2c04-435d-b909-ad97a75c32b8-kube-api-access-nqmt2\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627168 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-podres\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627168 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-proc\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627168 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-sys\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627168 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-lib-modules\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627362 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-proc\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627362 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-sys\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627362 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-lib-modules\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.627362 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.627263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/448b1272-2c04-435d-b909-ad97a75c32b8-podres\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.635374 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.635351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmt2\" (UniqueName: \"kubernetes.io/projected/448b1272-2c04-435d-b909-ad97a75c32b8-kube-api-access-nqmt2\") pod \"perf-node-gather-daemonset-zfnrb\" (UID: \"448b1272-2c04-435d-b909-ad97a75c32b8\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:22.762830 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:22.762746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:23.031928 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:23.031898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"]
Sep 16 14:07:23.034266 ip-10-0-137-128 kubenswrapper[2579]: W0916 14:07:23.034234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod448b1272_2c04_435d_b909_ad97a75c32b8.slice/crio-3a01bb44df6588f947893e5e62f7845591907e13d6013104a5c16f5f3f2ab604 WatchSource:0}: Error finding container 3a01bb44df6588f947893e5e62f7845591907e13d6013104a5c16f5f3f2ab604: Status 404 returned error can't find the container with id 3a01bb44df6588f947893e5e62f7845591907e13d6013104a5c16f5f3f2ab604
Sep 16 14:07:23.084118 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:23.081547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb" event={"ID":"448b1272-2c04-435d-b909-ad97a75c32b8","Type":"ContainerStarted","Data":"3a01bb44df6588f947893e5e62f7845591907e13d6013104a5c16f5f3f2ab604"}
Sep 16 14:07:23.142257 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:23.142226    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/1.log"
Sep 16 14:07:23.147574 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:23.147543    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7ljz6_92004b1b-d2d8-413a-be62-9def7a380328/console-operator/2.log"
Sep 16 14:07:24.087235 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:24.087194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb" event={"ID":"448b1272-2c04-435d-b909-ad97a75c32b8","Type":"ContainerStarted","Data":"8883af3f64e591c877ee71bf584f05ab01c44d88a557faef44850a700f117871"}
Sep 16 14:07:24.087733 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:24.087290    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:24.106544 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:24.106494    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb" podStartSLOduration=2.106480281 podStartE2EDuration="2.106480281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:24.104660707 +0000 UTC m=+4750.143459829" watchObservedRunningTime="2026-09-16 14:07:24.106480281 +0000 UTC m=+4750.145279404"
Sep 16 14:07:25.269406 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:25.269373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cm276_0c3bcc27-8289-4ce0-896f-f530c2a30fba/dns/0.log"
Sep 16 14:07:25.295255 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:25.295216    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cm276_0c3bcc27-8289-4ce0-896f-f530c2a30fba/kube-rbac-proxy/0.log"
Sep 16 14:07:25.441844 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:25.441811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c9wd8_d218c9f2-9a63-4429-84b7-67d6d88169b9/dns-node-resolver/0.log"
Sep 16 14:07:26.005521 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:26.005485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68fd47758c-jwdmd_18890be2-3926-46fd-bb8b-43e68f22b2ca/registry/0.log"
Sep 16 14:07:26.029926 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:26.029894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8t554_9c704c5a-edcd-423d-831d-2cef17a01ecd/node-ca/0.log"
Sep 16 14:07:27.137399 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:27.137362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/0.log"
Sep 16 14:07:27.150522 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:27.150496    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/1.log"
Sep 16 14:07:27.180098 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:27.180071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 14:07:27.706447 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:27.706412    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jldt8_a07394e1-5dc3-4faa-a8ec-517ab84483e2/serve-healthcheck-canary/0.log"
Sep 16 14:07:28.322527 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:28.322500    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rcht6_cce6fb50-b915-40c7-a3d3-d934cbfaadc4/kube-rbac-proxy/0.log"
Sep 16 14:07:28.344288 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:28.344262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rcht6_cce6fb50-b915-40c7-a3d3-d934cbfaadc4/exporter/0.log"
Sep 16 14:07:28.365476 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:28.365454    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rcht6_cce6fb50-b915-40c7-a3d3-d934cbfaadc4/extractor/0.log"
Sep 16 14:07:30.101221 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:30.101194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-zfnrb"
Sep 16 14:07:31.405319 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:31.405285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cc7b8d69b-h629m_e742f57c-4756-49fc-a306-2d69e9107678/manager/0.log"
Sep 16 14:07:32.114000 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:32.113969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/0.log"
Sep 16 14:07:32.126497 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:32.126455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/1.log"
Sep 16 14:07:32.143383 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:32.143346    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 14:07:33.599252 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:33.599215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/0.log"
Sep 16 14:07:33.613078 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:33.613056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9dkxq_2002c1fc-36a5-4cf5-bb6a-7c21fc1bbb75/istio-proxy/1.log"
Sep 16 14:07:33.628994 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:33.628974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55fccc44b8-9v5dg_c671399e-37d1-4307-8c61-a9464e26f327/router/0.log"
Sep 16 14:07:40.528690 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.528656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/kube-multus-additional-cni-plugins/0.log"
Sep 16 14:07:40.550083 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.550049    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/egress-router-binary-copy/0.log"
Sep 16 14:07:40.570331 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.570299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/cni-plugins/0.log"
Sep 16 14:07:40.593939 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.593911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/bond-cni-plugin/0.log"
Sep 16 14:07:40.613575 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.613520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/routeoverride-cni/0.log"
Sep 16 14:07:40.635384 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.635340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/whereabouts-cni-bincopy/0.log"
Sep 16 14:07:40.657818 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.657798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qvkp_4e8e3fb1-6e8f-4c8a-bb1a-15a59a9b0773/whereabouts-cni/0.log"
Sep 16 14:07:40.873595 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.873494    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mmr78_0e8cc591-1a31-41e9-b2e8-3f8e66aa2d45/kube-multus/0.log"
Sep 16 14:07:40.966472 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.966430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-57zxt_5487dc67-8a7e-4ff9-8a18-85d8f102b01e/network-metrics-daemon/0.log"
Sep 16 14:07:40.985028 ip-10-0-137-128 kubenswrapper[2579]: I0916 14:07:40.984992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-57zxt_5487dc67-8a7e-4ff9-8a18-85d8f102b01e/kube-rbac-proxy/0.log"