Sep 18 21:50:58.357643 ip-10-0-141-13 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 21:51:01.726033 ip-10-0-141-13 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:51:01.726033 ip-10-0-141-13 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 21:51:01.726033 ip-10-0-141-13 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:51:01.726033 ip-10-0-141-13 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 21:51:01.726033 ip-10-0-141-13 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:51:01.726806 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.726467    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 21:51:01.730972 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730957    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.730972 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730972    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730976    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730979    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730983    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730986    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730989    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730992    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730995    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.730998    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731001    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731003    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731006    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731009    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731011    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731014    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731016    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731019    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731022    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731025    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731028    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.731037 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731031    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731033    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731036    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731039    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731041    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731044    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731047    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731050    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731053    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731055    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731058    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731062    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731066    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731070    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731073    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731076    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731079    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731081    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731084    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.731537 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731086    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731089    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731091    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731094    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731096    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731099    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731101    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731103    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731111    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731115    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731118    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731120    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731123    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731126    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731129    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731132    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731134    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731137    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731139    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.731996 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731142    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731144    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731147    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731149    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731152    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731155    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731157    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731160    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731163    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731165    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731168    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731170    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731173    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731176    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731178    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731181    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731183    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731186    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731189    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731192    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.732494 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731194    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731197    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731200    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731202    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731206    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731208    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731211    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731214    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731217    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731219    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731222    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731224    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731227    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731229    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731232    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731234    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731237    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731241    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731244    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.732975 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731924    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731930    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731934    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731937    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731940    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731943    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731946    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731949    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731953    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731957    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731960    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731963    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731965    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731968    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731971    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731976    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731979    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731983    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731986    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.733486 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731989    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731992    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731995    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.731997    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732000    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732004    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732006    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732009    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732012    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732014    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732017    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732020    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732022    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732025    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732028    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732030    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732033    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732036    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732038    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732041    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.733959 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732043    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732046    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732048    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732051    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732053    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732056    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732059    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732061    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732064    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732066    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732070    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732072    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732075    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732078    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732080    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732083    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732085    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732088    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732090    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732093    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.734454 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732097    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732100    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732103    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732110    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732113    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732116    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732119    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732121    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732124    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732128    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732130    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732133    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732136    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732139    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732141    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732144    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732147    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732150    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732153    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.734944 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732155    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732158    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732161    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732164    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732166    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732170    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732172    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732175    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732177    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732180    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732183    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732185    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732188    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732190    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732193    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732195    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732198    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732200    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732203    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732205    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.735463 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732570    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732576    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732578    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732581    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732585    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732589    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732594    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732598    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732600    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732603    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732606    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732609    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732612    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732615    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732618    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732621    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732623    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732626    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732629    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.735951 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732631    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732634    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732636    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732639    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732641    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732644    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732646    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732649    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732651    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732654    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732657    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732659    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732662    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732665    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732667    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732671    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732674    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732676    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732679    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.736427 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732683    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732686    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732688    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732691    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732693    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732696    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732698    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732701    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732704    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732706    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732709    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732711    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732714    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732716    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732720    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732724    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732727    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732730    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732733    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.736894 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732736    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732739    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732741    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732744    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732747    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732750    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732752    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732755    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732757    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732760    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732762    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732767    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732771    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732773    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732776    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732778    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732781    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732784    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732786    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.737348 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732789    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732791    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732794    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732797    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732799    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732804    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732806    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732809    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732811    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732816    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732819    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732821    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732824    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732826    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732828    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732831    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732834    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732837    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.737881 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732840    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.732843    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732929    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732965    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732973    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732982    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732986    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732989    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732993    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.732998    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733001    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733003    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733007    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733010    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733013    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733016    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733020    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733023    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733026    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733029    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733032    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733035    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733038    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733041    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 21:51:01.738415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733045    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733048    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733054    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733057    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733060    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733063    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733066    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733069    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733072    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733075    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733079    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733082    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733085    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733089    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733092    2563 flags.go:64] FLAG: --enable-server="true"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733095    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733099    2563 flags.go:64] FLAG: --event-burst="100"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733102    2563 flags.go:64] FLAG: --event-qps="50"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733105    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733108    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733111    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733115    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733118    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733121    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733123    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 18 21:51:01.738998 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733127    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733130    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733133    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733136    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733139    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733141    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733144    2563 flags.go:64] FLAG: --feature-gates=""
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733148    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733151    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733155    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733158    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733161    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733164    2563 flags.go:64] FLAG: --help="false"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733167    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733170    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733173    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733175    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733179    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733182    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733185    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733189    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733192    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733195    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 21:51:01.739593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733198    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733201    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733204    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733206    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733209    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733212    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733215    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733218    2563 flags.go:64] FLAG: --lock-file=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733221    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733223    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733227    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733232    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733235    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733238    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733240    2563 flags.go:64] FLAG: --logging-format="text"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733243    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733246    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733249    2563 flags.go:64] FLAG: --manifest-url=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733253    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733257    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733260    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733264    2563 flags.go:64] FLAG: --max-pods="110"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733267    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733270    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733273    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 21:51:01.740196 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733276    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733279    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733282    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733285    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733293    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733296    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733299    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733302    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733305    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733310    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733313    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733316    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733319    2563 flags.go:64] FLAG: --port="10250"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733322    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733325    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7c3311315e9623d"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733328    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733331    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733334    2563 flags.go:64] FLAG: --register-node="true"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733337    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733341    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733344    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733346    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733349    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733353    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733356    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 21:51:01.740856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733358    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733362    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733365    2563 flags.go:64] FLAG: --runonce="false"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733368    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733371    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733374    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733377    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733380    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733383    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733386    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733389    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733405    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733408    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733411    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733414    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733416    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733419    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733422    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733427    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733430    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733433    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733437    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733439    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733442    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733445    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 21:51:01.741500 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733448    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733451    2563 flags.go:64] FLAG: --v="2"
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733455    2563 flags.go:64] FLAG: --version="false"
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733459    2563 flags.go:64] FLAG: --vmodule=""
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733462    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.733466    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733577    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733581    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733585    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733588    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733592    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733594    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733597    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733600    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733602    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733605    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733608    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733611    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733614    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733617    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733620    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.742284 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733624    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733627    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733630    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733633    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733636    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733639    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733641    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733644    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733647    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733650    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733653    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733657    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733661    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733666    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733669    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733671    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733674    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733676    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733679    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733683    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.743138 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733686    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733689    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733691    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733695    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733697    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733700    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733702    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733705    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733707    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733710    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733713    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733716    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733720    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733724    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733727    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733730    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733732    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733735    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733738    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.743984 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733758    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733762    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733765    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733768    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733771    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733774    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733777    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733780    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733783    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733786    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733789    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733791    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733795    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733798    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733800    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733803    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733806    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733809    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733811    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733815    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.744849 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733818    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733821    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733824    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733827    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733830    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733833    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733836    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733839    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733842    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733844    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733847    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733850    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733852    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733855    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733858    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733861    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733863    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733866    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733869    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733871    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.745775 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733874    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733877    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733880    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733882    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.733886    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.734343    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.744057    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.744076    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.744101    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.744115    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744205    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744212    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744219    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744224    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744229    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.746501 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744233    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744238    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744242    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744246    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744251    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744255    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744259    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744263    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744267    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744271    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744276    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744280    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744284    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744289    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744294    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744298    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744302    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744306    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744315    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.747206 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744320    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744325    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744329    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744333    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744338    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744343    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744347    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744351    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744355    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744360    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744364    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744368    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744372    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744376    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744381    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744385    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744389    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744409    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744414    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744418    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.747857 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744423    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744427    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744433    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744439    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744444    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744449    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744453    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744457    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744461    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744473    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744478    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744483    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744487    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744494    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744500    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744506    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744512    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744517    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744521    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.748515 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744526    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744530    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744534    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744538    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744542    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744547    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744551    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744556    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744560    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744564    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744568    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744572    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744577    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744581    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744586    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744603    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744607    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744611    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744615    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.748982 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744619    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744623    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744628    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744632    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744636    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744640    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744644    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744648    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744653    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744657    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744661    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744666    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744670    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744674    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744678    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.749511 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744683    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.744690    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744883    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744895    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744902    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744906    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744912    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744921    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744926    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744931    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744936    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744941    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744946    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744961    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:51:01.749914 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744965    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744969    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744973    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744977    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744981    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744985    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744990    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744994    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.744998    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745002    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745006    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745010    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745015    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745019    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745023    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745027    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745032    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745036    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745040    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:51:01.750302 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745044    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745048    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745052    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745056    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745061    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745065    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745071    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745076    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745080    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745084    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745089    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745093    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745097    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745109    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745114    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745118    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745122    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745126    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745130    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745134    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:51:01.750803 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745138    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745144    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745150    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745155    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745160    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745164    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745169    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745173    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745177    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745181    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745185    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745189    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745193    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745197    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745202    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745206    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745210    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745214    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745218    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:51:01.751460 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745223    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745227    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745231    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745235    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745239    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745244    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745248    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745260    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745264    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745268    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745272    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745276    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745280    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745284    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745288    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745292    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745296    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745300    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745304    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745309    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:51:01.752236 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745313    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745317    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745321    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745326    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745330    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745334    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745338    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:01.745342    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.745350    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.745502    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.748145    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.749180    2563 server.go:1019] "Starting client certificate rotation"
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.749280    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:51:01.753087 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.749312    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:51:01.753644 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.751738    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:51:01.753644 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.753490    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:51:01.758233 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.758219    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 18 21:51:01.760590 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.760578    2563 log.go:25] "Validated CRI v1 image API"
Sep 18 21:51:01.761626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.761605    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 21:51:01.764415 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.764374    2563 fs.go:135] Filesystem UUIDs: map[3c6f5b54-890e-42ed-82cc-3a7e502d9c50:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 84d2dfad-3058-428c-a79a-1c6327707c35:/dev/nvme0n1p4]
Sep 18 21:51:01.764498 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.764415    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 21:51:01.768459 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.768335    2563 manager.go:216] Machine: {Timestamp:2026-09-18 21:51:01.76789079 +0000 UTC m=+1.966599080 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100379 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2713edb6d0570a3db2cc81162f1040 SystemUUID:ec2713ed-b6d0-570a-3db2-cc81162f1040 BootID:0c3591d0-f243-4e0c-9692-f0cfadf33a6b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fe:aa:45:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fe:aa:45:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:ae:08:16:3a:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 21:51:01.768459 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.768456    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 21:51:01.768583 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.768562    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 21:51:01.770021 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.769992    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 21:51:01.770144 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.770026    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 21:51:01.771033 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.771023    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 21:51:01.771067 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.771036    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 21:51:01.771067 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.771044    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:51:01.771067 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.771057    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:51:01.771152 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.771118    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 21:51:01.774471 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774454    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:51:01.774522 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774494    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:51:01.774635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774625    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 21:51:01.774666 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774653    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 21:51:01.774666 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774664    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 21:51:01.774720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774675    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 21:51:01.774720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774684    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 18 21:51:01.774720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.774692    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 21:51:01.776026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.775998    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:51:01.776133 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.776033    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:51:01.778737 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.778719    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 21:51:01.780626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.780609    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 21:51:01.780714 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.780647    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 21:51:01.781624 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781611    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781631    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781640    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781649    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781661    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781669    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781688    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 21:51:01.781700 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781696    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781706    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781714    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781726    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781734    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781760    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 21:51:01.781934 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.781768    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 21:51:01.785278 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.785263    2563 server.go:1268] "Started kubelet"
Sep 18 21:51:01.785351 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.785306    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 21:51:01.785422 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.785339    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 21:51:01.785422 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.785418    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 21:51:01.785976 ip-10-0-141-13 systemd[1]: Started Kubernetes Kubelet.
Sep 18 21:51:01.786707 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.786548    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 21:51:01.788932 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.788915    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 21:51:01.790933 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.790909    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 21:51:01.790933 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.790926    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-13.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 21:51:01.791061 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.790985    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 21:51:01.791417 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.791368    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 21:51:01.792976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.792963    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 21:51:01.793044 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.792976    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 21:51:01.793283 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.793247    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:01.793735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793576    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 21:51:01.793735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793590    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 21:51:01.793735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793607    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 21:51:01.793735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793660    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 21:51:01.793735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793670    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 21:51:01.794054 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793893    2563 factory.go:55] Registering systemd factory
Sep 18 21:51:01.794054 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.793905    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 18 21:51:01.794141 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794087    2563 factory.go:153] Registering CRI-O factory
Sep 18 21:51:01.794141 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794099    2563 factory.go:223] Registration of the crio container factory successfully
Sep 18 21:51:01.794244 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794148    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 21:51:01.794244 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794176    2563 factory.go:103] Registering Raw factory
Sep 18 21:51:01.794244 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794188    2563 manager.go:1196] Started watching for new ooms in manager
Sep 18 21:51:01.794714 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.794691    2563 manager.go:349] Starting recovery of all containers
Sep 18 21:51:01.795528 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.795508    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 21:51:01.795803 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.795662    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 21:51:01.816945 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.816887    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdgm5"
Sep 18 21:51:01.817029 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.816970    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zdgm5"
Sep 18 21:51:01.824251 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.824233    2563 manager.go:354] Recovery completed
Sep 18 21:51:01.828127 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.828113    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.830005 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.829987    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.830094 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830018    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.830094 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830033    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.830557 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830545    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 21:51:01.830557 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830551    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.830557 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830556    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 21:51:01.830721 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.830573    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:51:01.831271 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.831259    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.831321 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.831285    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.831321 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.831298    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.832605 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.832590    2563 policy_none.go:49] "None policy: Start"
Sep 18 21:51:01.832653 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.832608    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 21:51:01.832653 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.832619    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 21:51:01.834569 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.834554    2563 policy_none.go:47] "Start"
Sep 18 21:51:01.869130 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869022    2563 manager.go:337] "Starting Device Plugin manager"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.869158    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869171    2563 server.go:87] "Starting device plugin registration server"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869458    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869483    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869563    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869646    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.869656    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.870090    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.870132    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.874704    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.875823    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.875858    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.875875    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.875919    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 21:51:01.885938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.878360    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:01.969940 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.969922    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.970673 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.970659    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.970738 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.970687    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.970738 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.970698    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.970738 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.970720    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.976333 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.976298    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"]
Sep 18 21:51:01.976376 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.976352    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.976999 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.976987    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.977051 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.977011    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.977051 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.977021    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.979020 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979009    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.979151 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.979233 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979166    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.979684 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979667    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.979760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979686    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.979760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979699    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.979760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979702    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.979760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979747    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.979760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979714    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.980007 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.979998    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.980043 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.980012    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-13.ec2.internal\": node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:01.981830 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.981812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.981880 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.981846    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:51:01.988741 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.988728    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:51:01.988824 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.988752    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:51:01.988824 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.988761    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:51:01.995203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.995188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.995250 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.995212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ea39be3926c31923b8b3abafa3a95cc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-13.ec2.internal\" (UID: \"0ea39be3926c31923b8b3abafa3a95cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.995250 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:01.995226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.995951 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.995935    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-13.ec2.internal\" not found" node="ip-10-0-141-13.ec2.internal"
Sep 18 21:51:01.997337 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:01.997321    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.001848 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.001833    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-13.ec2.internal\" not found" node="ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095879 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ea39be3926c31923b8b3abafa3a95cc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-13.ec2.internal\" (UID: \"0ea39be3926c31923b8b3abafa3a95cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0ea39be3926c31923b8b3abafa3a95cc-config\") pod \"kube-apiserver-proxy-ip-10-0-141-13.ec2.internal\" (UID: \"0ea39be3926c31923b8b3abafa3a95cc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.095990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.095957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5473a0bcaf13d280266d1d7391deb31f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal\" (UID: \"5473a0bcaf13d280266d1d7391deb31f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.097964 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.097950    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.198291 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.198250    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.298829 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.298760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.298933 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.298766    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.305536 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.305517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:02.399753 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.399727    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.500213 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.500191    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.600738 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.600708    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.698536 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.698507    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:02.701437 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.701419    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.701503 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.701440    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:02.748939 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.748918    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 21:51:02.749329 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.749061    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:51:02.749329 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.749078    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:51:02.749329 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.749092    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:51:02.801516 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.801486    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.823457 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.823419    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:46:01 +0000 UTC" deadline="2028-05-07 11:38:34.476804939 +0000 UTC"
Sep 18 21:51:02.823457 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.823450    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14317h47m31.653357724s"
Sep 18 21:51:02.901346 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:02.901298    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5473a0bcaf13d280266d1d7391deb31f.slice/crio-1b1bf7f76b485454fedd078044993e81a7c3ccd1dae5755120d7429d0db95903 WatchSource:0}: Error finding container 1b1bf7f76b485454fedd078044993e81a7c3ccd1dae5755120d7429d0db95903: Status 404 returned error can't find the container with id 1b1bf7f76b485454fedd078044993e81a7c3ccd1dae5755120d7429d0db95903
Sep 18 21:51:02.901566 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:02.901546    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-13.ec2.internal\" not found"
Sep 18 21:51:02.901764 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:02.901742    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea39be3926c31923b8b3abafa3a95cc.slice/crio-842756ca4aeeea81e9ff36bf1da7465f4e9ded8f27956990a6dd6f7fe97f8b61 WatchSource:0}: Error finding container 842756ca4aeeea81e9ff36bf1da7465f4e9ded8f27956990a6dd6f7fe97f8b61: Status 404 returned error can't find the container with id 842756ca4aeeea81e9ff36bf1da7465f4e9ded8f27956990a6dd6f7fe97f8b61
Sep 18 21:51:02.906291 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.906274    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:51:02.995596 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:02.995572    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:03.093673 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.093645    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:03.131841 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.131788    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:51:03.132812 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.132799    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal"
Sep 18 21:51:03.146428 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.146411    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:51:03.776008 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.775985    2563 apiserver.go:52] "Watching apiserver"
Sep 18 21:51:03.785912 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.785888    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 21:51:03.786236 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.786212    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sdwt8","kube-system/konnectivity-agent-tpwpt","openshift-dns/node-resolver-p9gnp","openshift-image-registry/node-ca-x8w62","openshift-multus/multus-additional-cni-plugins-m2pkz","openshift-network-diagnostics/network-check-target-6bl8g","openshift-network-operator/iptables-alerter-cphgh","kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb","openshift-cluster-node-tuning-operator/tuned-q2xgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal","openshift-multus/multus-slc8t","openshift-multus/network-metrics-daemon-wpmw9","openshift-ovn-kubernetes/ovnkube-node-dvlk7"]
Sep 18 21:51:03.788634 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.788603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.790648 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.790623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.791323 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.791305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.791796 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.791781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-txkr2\""
Sep 18 21:51:03.792036 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.792020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 21:51:03.792151 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.792136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 21:51:03.792458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.792441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 21:51:03.792635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.792612    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.792766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.792747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.795230 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.793760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 21:51:03.795230 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.794556    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 21:51:03.795673 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.795519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4m67k\""
Sep 18 21:51:03.796425 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.796407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.797065 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.797048    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 21:51:03.798599 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.798583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:03.799257 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.799241    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.799257 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.799252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.799456 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.799440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.799502 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.799469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bxpfh\""
Sep 18 21:51:03.800769 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.800750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:03.800872 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:03.800827    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:03.802979 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.802962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:03.803069 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:03.803036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:03.803803 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-cnibin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.803892 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-conf-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.803892 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8966\" (UniqueName: \"kubernetes.io/projected/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-kube-api-access-n8966\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.803892 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804041 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804041 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-system-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804041 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.803995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-netns\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804041 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-etc-kubernetes\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/379711c8-e41a-48ed-9eb5-594d76376be9-kube-api-access-grzmd\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgvj\" (UniqueName: \"kubernetes.io/projected/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-kube-api-access-wmgvj\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-serviceca\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-system-cni-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-multus\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-host-slash\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-iptables-alerter-script\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5hp\" (UniqueName: \"kubernetes.io/projected/625462d5-1360-4c58-9b03-2aee7bdebf98-kube-api-access-5j5hp\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-cni-binary-copy\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-socket-dir-parent\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-k8s-cni-cncf-io\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-kubelet\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804432 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-hostroot\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-bin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-multus-daemon-config\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-cnibin\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.804720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-multus-certs\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.804720 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.804563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-host\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.805210 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.805194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.807498 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.807475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.807590 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.807562    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 21:51:03.808515 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808476    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.808613 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808505    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.808684 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 21:51:03.808739 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.808881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xq7lr\""
Sep 18 21:51:03.808881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4g64n\""
Sep 18 21:51:03.809000 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.808957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.809735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.809687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:03.811573 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.811552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.811690 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.811588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g6tcr\""
Sep 18 21:51:03.811772 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.811714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5h7cd\""
Sep 18 21:51:03.811841 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.811779    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.811841 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.811800    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.812101 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.812070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:03.812179 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:03.812136    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:03.814051 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.814034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 21:51:03.814151 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.814084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-999qd\""
Sep 18 21:51:03.814305 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.814282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.814383 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.814334    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 21:51:03.818489 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.818463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 21:51:03.818633 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.818608    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 21:51:03.818754 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.818730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 21:51:03.818754 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.818748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 21:51:03.833988 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.833205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lp4hd\""
Sep 18 21:51:03.833988 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.833896    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 21:51:03.835220 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.835199    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 21:51:03.847869 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.847850    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:51:03.864133 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.864110    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zqwbg"
Sep 18 21:51:03.880115 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.880074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal" event={"ID":"5473a0bcaf13d280266d1d7391deb31f","Type":"ContainerStarted","Data":"1b1bf7f76b485454fedd078044993e81a7c3ccd1dae5755120d7429d0db95903"}
Sep 18 21:51:03.881286 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.881265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal" event={"ID":"0ea39be3926c31923b8b3abafa3a95cc","Type":"ContainerStarted","Data":"842756ca4aeeea81e9ff36bf1da7465f4e9ded8f27956990a6dd6f7fe97f8b61"}
Sep 18 21:51:03.883691 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.883668    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zqwbg"
Sep 18 21:51:03.894733 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.894717    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 21:51:03.905519 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-bin\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.905626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-env-overrides\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.905626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-system-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-netns\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-etc-kubernetes\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905626 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-etc-kubernetes\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-system-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-netns\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/379711c8-e41a-48ed-9eb5-594d76376be9-kube-api-access-grzmd\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgvj\" (UniqueName: \"kubernetes.io/projected/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-kube-api-access-wmgvj\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-device-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc48f62-0f58-452d-ba39-5b456370065a-agent-certs\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-kubelet\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-log-socket\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.905881 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-config\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-system-cni-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-host-slash\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-system-cni-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysconfig\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.905995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-host-slash\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-cni-binary-copy\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a42efa-8572-4d8a-817a-f8d6542f0f4d-tmp-dir\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-dbus\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91576d05-55c7-41e0-925e-1d2573f17265-ovn-node-metrics-cert\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-conf\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-multus-daemon-config\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-cni-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.906350 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-socket-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-systemd\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-node-log\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-cnibin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-conf-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxcd\" (UniqueName: \"kubernetes.io/projected/02a42efa-8572-4d8a-817a-f8d6542f0f4d-kube-api-access-dfxcd\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-cnibin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-registration-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-conf-dir\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-kubernetes\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-tuned\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-tmp\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-cni-binary-copy\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-serviceca\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-var-lib-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/379711c8-e41a-48ed-9eb5-594d76376be9-multus-daemon-config\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-netd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-modprobe-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-sys\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-lib-modules\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-multus\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-multus\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.906983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc48f62-0f58-452d-ba39-5b456370065a-konnectivity-ca\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/625462d5-1360-4c58-9b03-2aee7bdebf98-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.907894 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-iptables-alerter-script\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5hp\" (UniqueName: \"kubernetes.io/projected/625462d5-1360-4c58-9b03-2aee7bdebf98-kube-api-access-5j5hp\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-serviceca\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-socket-dir-parent\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-k8s-cni-cncf-io\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-kubelet\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-multus-socket-dir-parent\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-run\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-slash\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-k8s-cni-cncf-io\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-netns\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-hostroot\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-kubelet\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-etc-selinux\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4dkl\" (UniqueName: \"kubernetes.io/projected/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kube-api-access-p4dkl\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-hostroot\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:03.908701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnzb\" (UniqueName: \"kubernetes.io/projected/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-kube-api-access-8gnzb\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-etc-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-bin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02a42efa-8572-4d8a-817a-f8d6542f0f4d-hosts-file\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-var-lib-cni-bin\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-kubelet-config\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-sys-fs\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-iptables-alerter-script\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-systemd-units\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbkv\" (UniqueName: \"kubernetes.io/projected/91576d05-55c7-41e0-925e-1d2573f17265-kube-api-access-mbbkv\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-cnibin\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-multus-certs\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-host\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/625462d5-1360-4c58-9b03-2aee7bdebf98-cnibin\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/379711c8-e41a-48ed-9eb5-594d76376be9-host-run-multus-certs\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-host\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.909478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-host\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jm8g\" (UniqueName: \"kubernetes.io/projected/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-kube-api-access-6jm8g\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-ovn\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-script-lib\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8966\" (UniqueName: \"kubernetes.io/projected/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-kube-api-access-n8966\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-var-lib-kubelet\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:03.910258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.907902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-systemd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:03.927546 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.927525    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 21:51:03.931019 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.931000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8966\" (UniqueName: \"kubernetes.io/projected/7f5e819a-8e56-41f7-ab2a-0d9f8a382813-kube-api-access-n8966\") pod \"node-ca-x8w62\" (UID: \"7f5e819a-8e56-41f7-ab2a-0d9f8a382813\") " pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:03.931096 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.931060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5hp\" (UniqueName: \"kubernetes.io/projected/625462d5-1360-4c58-9b03-2aee7bdebf98-kube-api-access-5j5hp\") pod \"multus-additional-cni-plugins-m2pkz\" (UID: \"625462d5-1360-4c58-9b03-2aee7bdebf98\") " pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:03.944437 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.944414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgvj\" (UniqueName: \"kubernetes.io/projected/5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c-kube-api-access-wmgvj\") pod \"iptables-alerter-cphgh\" (UID: \"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c\") " pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:03.944536 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:03.944516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzmd\" (UniqueName: \"kubernetes.io/projected/379711c8-e41a-48ed-9eb5-594d76376be9-kube-api-access-grzmd\") pod \"multus-slc8t\" (UID: \"379711c8-e41a-48ed-9eb5-594d76376be9\") " pod="openshift-multus/multus-slc8t"
Sep 18 21:51:04.008153 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-kubelet-config\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.008153 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-sys-fs\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-systemd-units\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbkv\" (UniqueName: \"kubernetes.io/projected/91576d05-55c7-41e0-925e-1d2573f17265-kube-api-access-mbbkv\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-host\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jm8g\" (UniqueName: \"kubernetes.io/projected/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-kube-api-access-6jm8g\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-ovn\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-systemd-units\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-script-lib\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-host\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-var-lib-kubelet\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-sys-fs\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.008319 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-systemd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-kubelet-config\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-systemd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-var-lib-kubelet\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-ovn\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-bin\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-bin\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-env-overrides\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-device-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc48f62-0f58-452d-ba39-5b456370065a-agent-certs\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-kubelet\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.008680    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-log-socket\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-device-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-config\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.008754    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:04.508735491 +0000 UTC m=+4.707443787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:04.009106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-kubelet\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysconfig\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a42efa-8572-4d8a-817a-f8d6542f0f4d-tmp-dir\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-dbus\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91576d05-55c7-41e0-925e-1d2573f17265-ovn-node-metrics-cert\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-conf\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-script-lib\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.008999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-env-overrides\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-socket-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysconfig\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53076ba4-5cba-44f1-bf24-28f762dd893d-dbus\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-log-socket\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-systemd\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-node-log\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.009984 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxcd\" (UniqueName: \"kubernetes.io/projected/02a42efa-8572-4d8a-817a-f8d6542f0f4d-kube-api-access-dfxcd\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-conf\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a42efa-8572-4d8a-817a-f8d6542f0f4d-tmp-dir\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-node-log\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-systemd\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-socket-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-registration-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91576d05-55c7-41e0-925e-1d2573f17265-ovnkube-config\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-sysctl-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-kubernetes\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-tuned\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-registration-dir\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-kubernetes\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-tmp\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-var-lib-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-netd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.010827 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-modprobe-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-sys\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-lib-modules\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc48f62-0f58-452d-ba39-5b456370065a-konnectivity-ca\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-cni-netd\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-run\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-slash\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-modprobe-d\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-netns\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-etc-selinux\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dkl\" (UniqueName: \"kubernetes.io/projected/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kube-api-access-p4dkl\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-run-netns\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-lib-modules\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.011419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-var-lib-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-slash\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-run\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnzb\" (UniqueName: \"kubernetes.io/projected/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-kube-api-access-8gnzb\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-etc-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-sys\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.009943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02a42efa-8572-4d8a-817a-f8d6542f0f4d-hosts-file\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02a42efa-8572-4d8a-817a-f8d6542f0f4d-hosts-file\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.010086    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-run-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.010146    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:04.510127801 +0000 UTC m=+4.708836096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-etc-openvswitch\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91576d05-55c7-41e0-925e-1d2573f17265-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-etc-selinux\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.010254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bbc48f62-0f58-452d-ba39-5b456370065a-konnectivity-ca\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.011632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-tmp\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.012114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.011681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91576d05-55c7-41e0-925e-1d2573f17265-ovn-node-metrics-cert\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.012682 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.011697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bbc48f62-0f58-452d-ba39-5b456370065a-agent-certs\") pod \"konnectivity-agent-tpwpt\" (UID: \"bbc48f62-0f58-452d-ba39-5b456370065a\") " pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:04.012682 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.012009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-etc-tuned\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.027159 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.027106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxcd\" (UniqueName: \"kubernetes.io/projected/02a42efa-8572-4d8a-817a-f8d6542f0f4d-kube-api-access-dfxcd\") pod \"node-resolver-p9gnp\" (UID: \"02a42efa-8572-4d8a-817a-f8d6542f0f4d\") " pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.028819 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.028799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbkv\" (UniqueName: \"kubernetes.io/projected/91576d05-55c7-41e0-925e-1d2573f17265-kube-api-access-mbbkv\") pod \"ovnkube-node-dvlk7\" (UID: \"91576d05-55c7-41e0-925e-1d2573f17265\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.034860 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.034843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnzb\" (UniqueName: \"kubernetes.io/projected/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-kube-api-access-8gnzb\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:04.045679 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.045658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dkl\" (UniqueName: \"kubernetes.io/projected/ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039-kube-api-access-p4dkl\") pod \"aws-ebs-csi-driver-node-9bkqb\" (UID: \"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.045839 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.045821    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:04.045915 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.045844    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:04.045915 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.045854    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:04.045915 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.045907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:04.545893916 +0000 UTC m=+4.744602193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:04.046732 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.046714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jm8g\" (UniqueName: \"kubernetes.io/projected/7fcbe2dc-620f-4d63-914b-06cdbcd44b16-kube-api-access-6jm8g\") pod \"tuned-q2xgp\" (UID: \"7fcbe2dc-620f-4d63-914b-06cdbcd44b16\") " pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.101110 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.101084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2pkz"
Sep 18 21:51:04.106657 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.106637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-slc8t"
Sep 18 21:51:04.115493 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.115473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cphgh"
Sep 18 21:51:04.121310 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.121291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8w62"
Sep 18 21:51:04.126826 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.126807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9gnp"
Sep 18 21:51:04.135478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.135458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb"
Sep 18 21:51:04.142184 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.142167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q2xgp"
Sep 18 21:51:04.149848 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.149831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:04.153287 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.153272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:04.243449 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.243423    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:04.513342 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.513301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:04.513520 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.513362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:04.513520 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.513481    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:04.513520 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.513504    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:04.513638 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.513542    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:05.513524311 +0000 UTC m=+5.712232593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:04.513638 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.513563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:05.513549151 +0000 UTC m=+5.712257429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:04.602262 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.602235    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:04.613868 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.613841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:04.614009 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.613990    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:04.614064 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.614015    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:04.614064 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.614029    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:04.614128 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:04.614081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:05.614068739 +0000 UTC m=+5.812777016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:04.787972 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.787949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc48f62_0f58_452d_ba39_5b456370065a.slice/crio-277d8d978c71a98013abc80c323a015a1dfa022f3385103cbe205ed00796c003 WatchSource:0}: Error finding container 277d8d978c71a98013abc80c323a015a1dfa022f3385103cbe205ed00796c003: Status 404 returned error can't find the container with id 277d8d978c71a98013abc80c323a015a1dfa022f3385103cbe205ed00796c003
Sep 18 21:51:04.789253 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.789187    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e819a_8e56_41f7_ab2a_0d9f8a382813.slice/crio-3176018f10629b48396999da33a36f23067cc87df7270b4e0a8a80ca30fb70c2 WatchSource:0}: Error finding container 3176018f10629b48396999da33a36f23067cc87df7270b4e0a8a80ca30fb70c2: Status 404 returned error can't find the container with id 3176018f10629b48396999da33a36f23067cc87df7270b4e0a8a80ca30fb70c2
Sep 18 21:51:04.790428 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.790295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcbe2dc_620f_4d63_914b_06cdbcd44b16.slice/crio-d749b039faa4a7543ad4e03770097e0c7123c3f75fdcd0494c1034ac4fd0081c WatchSource:0}: Error finding container d749b039faa4a7543ad4e03770097e0c7123c3f75fdcd0494c1034ac4fd0081c: Status 404 returned error can't find the container with id d749b039faa4a7543ad4e03770097e0c7123c3f75fdcd0494c1034ac4fd0081c
Sep 18 21:51:04.793322 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.793302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625462d5_1360_4c58_9b03_2aee7bdebf98.slice/crio-ad28d16d06e3fa444d10103d1eb1088ba6ebc9c42b2cbcc886f340f71e75885e WatchSource:0}: Error finding container ad28d16d06e3fa444d10103d1eb1088ba6ebc9c42b2cbcc886f340f71e75885e: Status 404 returned error can't find the container with id ad28d16d06e3fa444d10103d1eb1088ba6ebc9c42b2cbcc886f340f71e75885e
Sep 18 21:51:04.794375 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.794357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec59bb63_ffcd_4ca6_8bfa_14c8ca41e039.slice/crio-fce18d62a8535794bf3f1f0832aeee34bd1fc1acd0ddd065d6958ab08c2e12b8 WatchSource:0}: Error finding container fce18d62a8535794bf3f1f0832aeee34bd1fc1acd0ddd065d6958ab08c2e12b8: Status 404 returned error can't find the container with id fce18d62a8535794bf3f1f0832aeee34bd1fc1acd0ddd065d6958ab08c2e12b8
Sep 18 21:51:04.795597 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.795513    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a42efa_8572_4d8a_817a_f8d6542f0f4d.slice/crio-881306d18fe266843317b9940d64336bffc08759963bcb66e881c80786f36535 WatchSource:0}: Error finding container 881306d18fe266843317b9940d64336bffc08759963bcb66e881c80786f36535: Status 404 returned error can't find the container with id 881306d18fe266843317b9940d64336bffc08759963bcb66e881c80786f36535
Sep 18 21:51:04.796382 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.796129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b535b5b_e3e5_4f31_a8e2_ca2e2469e20c.slice/crio-1f754fd61719a2b0718369088b12de22bdef1133eb6a358cf1fa1e4f53efafe1 WatchSource:0}: Error finding container 1f754fd61719a2b0718369088b12de22bdef1133eb6a358cf1fa1e4f53efafe1: Status 404 returned error can't find the container with id 1f754fd61719a2b0718369088b12de22bdef1133eb6a358cf1fa1e4f53efafe1
Sep 18 21:51:04.797068 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.797005    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91576d05_55c7_41e0_925e_1d2573f17265.slice/crio-bcc7f13458ed14d48fe49b9fe0c3cd7bfff5586ad4ede883955df5acdcc8dd78 WatchSource:0}: Error finding container bcc7f13458ed14d48fe49b9fe0c3cd7bfff5586ad4ede883955df5acdcc8dd78: Status 404 returned error can't find the container with id bcc7f13458ed14d48fe49b9fe0c3cd7bfff5586ad4ede883955df5acdcc8dd78
Sep 18 21:51:04.798589 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:04.798182    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379711c8_e41a_48ed_9eb5_594d76376be9.slice/crio-101a58cd6f29b275eea66e4299445b86cdd3ee9a475db899488c6241074e29df WatchSource:0}: Error finding container 101a58cd6f29b275eea66e4299445b86cdd3ee9a475db899488c6241074e29df: Status 404 returned error can't find the container with id 101a58cd6f29b275eea66e4299445b86cdd3ee9a475db899488c6241074e29df
Sep 18 21:51:04.885262 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.885166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal" event={"ID":"0ea39be3926c31923b8b3abafa3a95cc","Type":"ContainerStarted","Data":"d53e8ea0973196d0441140189fa3275b85835cb79b4118d6599290fc7085e138"}
Sep 18 21:51:04.886363 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.886337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9gnp" event={"ID":"02a42efa-8572-4d8a-817a-f8d6542f0f4d","Type":"ContainerStarted","Data":"881306d18fe266843317b9940d64336bffc08759963bcb66e881c80786f36535"}
Sep 18 21:51:04.886520 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.886431    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:46:03 +0000 UTC" deadline="2028-03-23 00:51:46.557970617 +0000 UTC"
Sep 18 21:51:04.886520 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.886454    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13227h0m41.671520471s"
Sep 18 21:51:04.887527 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.887470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb" event={"ID":"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039","Type":"ContainerStarted","Data":"fce18d62a8535794bf3f1f0832aeee34bd1fc1acd0ddd065d6958ab08c2e12b8"}
Sep 18 21:51:04.888776 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.888756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8w62" event={"ID":"7f5e819a-8e56-41f7-ab2a-0d9f8a382813","Type":"ContainerStarted","Data":"3176018f10629b48396999da33a36f23067cc87df7270b4e0a8a80ca30fb70c2"}
Sep 18 21:51:04.889935 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.889912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpwpt" event={"ID":"bbc48f62-0f58-452d-ba39-5b456370065a","Type":"ContainerStarted","Data":"277d8d978c71a98013abc80c323a015a1dfa022f3385103cbe205ed00796c003"}
Sep 18 21:51:04.891031 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.891012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slc8t" event={"ID":"379711c8-e41a-48ed-9eb5-594d76376be9","Type":"ContainerStarted","Data":"101a58cd6f29b275eea66e4299445b86cdd3ee9a475db899488c6241074e29df"}
Sep 18 21:51:04.892061 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.892040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"bcc7f13458ed14d48fe49b9fe0c3cd7bfff5586ad4ede883955df5acdcc8dd78"}
Sep 18 21:51:04.893283 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.893257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cphgh" event={"ID":"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c","Type":"ContainerStarted","Data":"1f754fd61719a2b0718369088b12de22bdef1133eb6a358cf1fa1e4f53efafe1"}
Sep 18 21:51:04.894383 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.894361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerStarted","Data":"ad28d16d06e3fa444d10103d1eb1088ba6ebc9c42b2cbcc886f340f71e75885e"}
Sep 18 21:51:04.895717 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.895697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q2xgp" event={"ID":"7fcbe2dc-620f-4d63-914b-06cdbcd44b16","Type":"ContainerStarted","Data":"d749b039faa4a7543ad4e03770097e0c7123c3f75fdcd0494c1034ac4fd0081c"}
Sep 18 21:51:04.907941 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:04.907898    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal" podStartSLOduration=1.907886198 podStartE2EDuration="1.907886198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:04.907474887 +0000 UTC m=+5.106183185" watchObservedRunningTime="2026-09-18 21:51:04.907886198 +0000 UTC m=+5.106594498"
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.520710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.520989    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.521031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.521061    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:07.521040923 +0000 UTC m=+7.719749206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.521129    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:05.521216 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.521181    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:07.521165453 +0000 UTC m=+7.719873735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:05.600711 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.600662    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:51:05.622249 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.622210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:05.622443 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.622417    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:05.622443 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.622436    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:05.622564 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.622447    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:05.622564 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.622499    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:07.622481442 +0000 UTC m=+7.821189720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:05.878497 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.878467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:05.878927 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.878581    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:05.879035 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.878992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:05.879111 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.879086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:05.879170 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.879154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:05.879265 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:05.879237    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:05.887505 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.887439    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:46:03 +0000 UTC" deadline="2028-06-03 01:52:39.868754072 +0000 UTC"
Sep 18 21:51:05.887505 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.887486    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14956h1m33.981271388s"
Sep 18 21:51:05.910076 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.910045    2563 generic.go:356] "Generic (PLEG): container finished" podID="5473a0bcaf13d280266d1d7391deb31f" containerID="10092b29a1986bfce3a411a8d2af0621085bf9ca57bf733dd68387e19a880283" exitCode=0
Sep 18 21:51:05.910226 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:05.910148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal" event={"ID":"5473a0bcaf13d280266d1d7391deb31f","Type":"ContainerDied","Data":"10092b29a1986bfce3a411a8d2af0621085bf9ca57bf733dd68387e19a880283"}
Sep 18 21:51:06.939846 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:06.939119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal" event={"ID":"5473a0bcaf13d280266d1d7391deb31f","Type":"ContainerStarted","Data":"d91c71aaf45e0caa65aa670d85dbfb713ff15c8b33f39419722e76a75b22d8ad"}
Sep 18 21:51:07.537143 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.537072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:07.537143 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.537155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:07.537496 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.537279    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:07.537496 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.537340    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:11.537321505 +0000 UTC m=+11.736029791 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:07.537496 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.537420    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:07.537496 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.537461    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:11.537447148 +0000 UTC m=+11.736155441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:07.638149 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.637466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:07.638149 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.637678    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:07.638149 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.637703    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:07.638149 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.637716    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:07.638149 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.637774    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:11.637756382 +0000 UTC m=+11.836464674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:07.878025 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.877612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:07.878025 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.877736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:07.878025 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.877611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:07.878025 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.877963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:07.878307 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:07.878098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:07.878307 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:07.878195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:09.877504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:09.877505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:09.877630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:09.877647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:09.877739    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:09.878016 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:09.877829    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:11.569678 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.569645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:11.570112 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.569702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:11.570112 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.569811    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:11.570112 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.569891    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:19.569870875 +0000 UTC m=+19.768579167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:11.570112 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.569818    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:11.570112 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.569981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:19.56996181 +0000 UTC m=+19.768670101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:11.670986 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.670875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:11.671160 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.671032    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:11.671160 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.671053    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:11.671160 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.671064    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:11.671160 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.671125    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:19.671106985 +0000 UTC m=+19.869815267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.878854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.878951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.878988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:11.879038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.879154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:11.879275 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:11.879235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:13.876678 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:13.876643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:13.877094 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:13.876791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:13.877094 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:13.876868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:13.877094 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:13.876785    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:13.877094 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:13.876907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:13.877094 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:13.877014    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:15.877068 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:15.877029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:15.877636 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:15.877075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:15.877636 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:15.877167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:15.877636 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:15.877230    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:15.877636 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:15.877272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:15.877636 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:15.877334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:17.876722 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:17.876687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:17.877114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:17.876731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:17.877114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:17.876696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:17.877114 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:17.876796    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:17.877114 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:17.876876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:17.877114 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:17.876952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:19.629243 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.629213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:19.629679 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.629265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:19.629679 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.629372    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:19.629679 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.629385    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:19.629679 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.629453    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:51:35.629431892 +0000 UTC m=+35.828140172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:19.629679 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.629472    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:35.629465993 +0000 UTC m=+35.828174270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:19.730333 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.730303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:19.730538 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.730516    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:19.730599 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.730548    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:19.730599 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.730561    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:19.730703 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.730620    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:35.730603998 +0000 UTC m=+35.929312274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:19.876887 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.876856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:19.877058 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.876891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:19.877058 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:19.876854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:19.877058 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.876975    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:19.877208 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.877080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:19.877208 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:19.877168    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:21.877602 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:21.877575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:21.877955 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:21.877726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:21.877955 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:21.877749    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:21.877955 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:21.877811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:21.877955 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:21.877823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:21.877955 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:21.877875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:22.970026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.969477    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="c3d8e2eb9ffc71c2e29d98064124c08840811491f6dccc73123c3982f8124d66" exitCode=0
Sep 18 21:51:22.970026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.969555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"c3d8e2eb9ffc71c2e29d98064124c08840811491f6dccc73123c3982f8124d66"}
Sep 18 21:51:22.972677 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.972618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q2xgp" event={"ID":"7fcbe2dc-620f-4d63-914b-06cdbcd44b16","Type":"ContainerStarted","Data":"c6bce9b6fecb7e8c4608744869e9f877fce97e213d756052f8f9b0ee28147fed"}
Sep 18 21:51:22.974306 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.974272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9gnp" event={"ID":"02a42efa-8572-4d8a-817a-f8d6542f0f4d","Type":"ContainerStarted","Data":"e2e813b5068b1b36ade40d6213890a9334f9d15db9031926bda168fd45314a56"}
Sep 18 21:51:22.976749 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.976723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb" event={"ID":"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039","Type":"ContainerStarted","Data":"75d95ac5450029aed834981f6c48abe76b13c8804b557e0203e9382bf35665c3"}
Sep 18 21:51:22.978449 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.978425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8w62" event={"ID":"7f5e819a-8e56-41f7-ab2a-0d9f8a382813","Type":"ContainerStarted","Data":"df0c7c1f296891b28f1e3e022b9e12842d63dee0e2bb7862478d38ea9aac163e"}
Sep 18 21:51:22.980245 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.980220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpwpt" event={"ID":"bbc48f62-0f58-452d-ba39-5b456370065a","Type":"ContainerStarted","Data":"3cb4e7a57113394f295fb3ecc364b6ab035bff34802ad0d2a2fd423d90d93efb"}
Sep 18 21:51:22.982374 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.982349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-slc8t" event={"ID":"379711c8-e41a-48ed-9eb5-594d76376be9","Type":"ContainerStarted","Data":"0328fdb09f93ed9f280530692a75fbfed4c9cdb251d0af68a2642b69696596f7"}
Sep 18 21:51:22.985954 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.985922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:51:22.986272 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986254    2563 generic.go:356] "Generic (PLEG): container finished" podID="91576d05-55c7-41e0-925e-1d2573f17265" containerID="d6c5cde734f0cf7b1807903bce3e8ecdbdf178e21c18572edf2d441eb922a208" exitCode=1
Sep 18 21:51:22.986342 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"a155472333578cbd613e29d73c4a17fa6d183482698fb691034f7aa36b4fd1e3"}
Sep 18 21:51:22.986342 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"6dd2868478d7be04ce9d14cf8c398f1bcafa7c9905388b497ba8d7d4bf09108b"}
Sep 18 21:51:22.986423 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"9bd91b2600b7b72f79f03203fb0c3ded176e2556ede8d137006019417e46cee2"}
Sep 18 21:51:22.986423 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"bbcb3cbe6106d3fe988364cde3a918e9438f8fee730621d714505894d8ea1d36"}
Sep 18 21:51:22.986552 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerDied","Data":"d6c5cde734f0cf7b1807903bce3e8ecdbdf178e21c18572edf2d441eb922a208"}
Sep 18 21:51:22.986552 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.986482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"27fbf429333b6783005112e5ff2c9a7e6fb485230f98c2c998b28ab617196e7d"}
Sep 18 21:51:22.998387 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:22.998340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-13.ec2.internal" podStartSLOduration=19.998324251 podStartE2EDuration="19.998324251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.968288713 +0000 UTC m=+7.166997013" watchObservedRunningTime="2026-09-18 21:51:22.998324251 +0000 UTC m=+23.197032553"
Sep 18 21:51:23.038115 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.038071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-slc8t" podStartSLOduration=4.706407242 podStartE2EDuration="22.03805793s" totalImagesPullingTime="17.331650688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:23.037935594 +0000 UTC m=+23.236643894" watchObservedRunningTime="2026-09-18 21:51:23.03805793 +0000 UTC m=+23.236766229"
Sep 18 21:51:23.061570 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.061529    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p9gnp" podStartSLOduration=3.774547568 podStartE2EDuration="21.061515555s" totalImagesPullingTime="17.286967987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:23.061168804 +0000 UTC m=+23.259877102" watchObservedRunningTime="2026-09-18 21:51:23.061515555 +0000 UTC m=+23.260223853"
Sep 18 21:51:23.114014 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.113977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x8w62" podStartSLOduration=3.848758734 podStartE2EDuration="21.113964955s" totalImagesPullingTime="17.265206221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:23.088526872 +0000 UTC m=+23.287235170" watchObservedRunningTime="2026-09-18 21:51:23.113964955 +0000 UTC m=+23.312673257"
Sep 18 21:51:23.141627 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.141592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q2xgp" podStartSLOduration=3.848327396 podStartE2EDuration="21.141579894s" totalImagesPullingTime="17.293252498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:23.113738803 +0000 UTC m=+23.312447101" watchObservedRunningTime="2026-09-18 21:51:23.141579894 +0000 UTC m=+23.340288193"
Sep 18 21:51:23.141977 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.141949    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tpwpt" podStartSLOduration=8.084429915 podStartE2EDuration="21.141942624s" totalImagesPullingTime="13.057512709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:23.141329856 +0000 UTC m=+23.340038155" watchObservedRunningTime="2026-09-18 21:51:23.141942624 +0000 UTC m=+23.340650923"
Sep 18 21:51:23.470575 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.470550    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 21:51:23.877063 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.876945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:23.877063 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.876976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:23.877063 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.876945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:23.877063 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:23.877057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:23.877451 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:23.877166    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:23.877451 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:23.877240    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:23.884721 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.884618    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T21:51:23.470572268Z","UUID":"01a67407-5558-48d2-a4e0-04047dca1cbd","Handler":null,"Name":"","Endpoint":""}
Sep 18 21:51:23.886241 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.886217    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 21:51:23.886362 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.886254    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 21:51:23.890355 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.890290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:23.890857 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.890838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:23.990524 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.990488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cphgh" event={"ID":"5b535b5b-e3e5-4f31-a8e2-ca2e2469e20c","Type":"ContainerStarted","Data":"0a4c1a1cec61b537d55a8ef7b8f56fbd4f56a904b69661150ee9c0848096c155"}
Sep 18 21:51:23.993336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.993306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb" event={"ID":"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039","Type":"ContainerStarted","Data":"5903443d7e6b5f3c2d6c289dd1dea7470c961883004abc8ca70edcbe3a6a5470"}
Sep 18 21:51:23.994362 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.994040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:23.994362 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:23.994326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tpwpt"
Sep 18 21:51:24.056608 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:24.056544    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cphgh" podStartSLOduration=4.770443902 podStartE2EDuration="22.056524371s" totalImagesPullingTime="17.286080469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:24.020528505 +0000 UTC m=+24.219236795" watchObservedRunningTime="2026-09-18 21:51:24.056524371 +0000 UTC m=+24.255232671"
Sep 18 21:51:24.998232 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:24.998190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb" event={"ID":"ec59bb63-ffcd-4ca6-8bfa-14c8ca41e039","Type":"ContainerStarted","Data":"cbb0e3cefcc09eae4aae4e82373e95c262f842bbaf154570975d74166d89ced1"}
Sep 18 21:51:25.023386 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:25.023349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9bkqb" podStartSLOduration=3.757548571 podStartE2EDuration="23.023330985s" totalImagesPullingTime="19.265782414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:25.022167351 +0000 UTC m=+25.220875650" watchObservedRunningTime="2026-09-18 21:51:25.023330985 +0000 UTC m=+25.222039330"
Sep 18 21:51:25.876582 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:25.876548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:25.876582 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:25.876571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:25.876806 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:25.876564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:25.876806 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:25.876684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:25.876806 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:25.876761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:25.876933 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:25.876849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:26.004100 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:26.004074    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:51:26.004552 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:26.004499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"b931604ffad32b8fd5bd9be37e6568a76b405ad9fdf61816fb7f67de8695c3a5"}
Sep 18 21:51:27.876989 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:27.876821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:27.877645 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:27.876826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:27.877645 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:27.876848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:27.877645 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:27.877139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:27.877645 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:27.877050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:27.877645 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:27.877220    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:28.012026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.012004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:51:28.012386 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.012370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"6e10d584de7fba95ea904f85a3d778ea3761e4c13a6bcea2d5f905355498f64a"}
Sep 18 21:51:28.012656 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.012633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:28.012828 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.012705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:28.012828 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.012770    2563 scope.go:117] "RemoveContainer" containerID="d6c5cde734f0cf7b1807903bce3e8ecdbdf178e21c18572edf2d441eb922a208"
Sep 18 21:51:28.014383 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.014358    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="1d7ed913e3b6bdca75d31af186bafe8ba475fbc9349b904cc4976252a1d809f1" exitCode=0
Sep 18 21:51:28.014495 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.014412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"1d7ed913e3b6bdca75d31af186bafe8ba475fbc9349b904cc4976252a1d809f1"}
Sep 18 21:51:28.027915 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:28.027896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:29.020988 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.020810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:51:29.021425 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.021381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" event={"ID":"91576d05-55c7-41e0-925e-1d2573f17265","Type":"ContainerStarted","Data":"01c0d35fd3d7b5630d3d22f8ef94583bea14092be8de9cd5df1a5226bf548f7c"}
Sep 18 21:51:29.021715 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.021700    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:29.035639 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.035620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:51:29.058677 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.058640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7" podStartSLOduration=9.759362792 podStartE2EDuration="27.058627683s" totalImagesPullingTime="17.299264891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:29.057466491 +0000 UTC m=+29.256174790" watchObservedRunningTime="2026-09-18 21:51:29.058627683 +0000 UTC m=+29.257335983"
Sep 18 21:51:29.413377 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.413348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sdwt8"]
Sep 18 21:51:29.413515 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.413489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:29.413603 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:29.413583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:29.415904 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.415881    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bl8g"]
Sep 18 21:51:29.416007 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.415980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:29.416090 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:29.416069    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:29.419106 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.419081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpmw9"]
Sep 18 21:51:29.419201 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:29.419168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:29.419259 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:29.419242    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:30.026009 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:30.025972    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="df6a9dca2315b02df9a33f29b68a5a9c89129d62ef745e7dec620ee3a4378064" exitCode=0
Sep 18 21:51:30.026493 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:30.026053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"df6a9dca2315b02df9a33f29b68a5a9c89129d62ef745e7dec620ee3a4378064"}
Sep 18 21:51:30.876986 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:30.876959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:30.877148 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:30.877046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:30.877148 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:30.877110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:30.877266 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:30.877216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:31.876928 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:31.876901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:31.877285 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:31.876994    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:32.032730 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:32.032704    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="31eb257864318f7bd22fdee47b8bb7d5ecbe2f7c41cc96da56bccdbef7db4c8b" exitCode=0
Sep 18 21:51:32.032822 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:32.032751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"31eb257864318f7bd22fdee47b8bb7d5ecbe2f7c41cc96da56bccdbef7db4c8b"}
Sep 18 21:51:32.876384 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:32.876209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:32.876559 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:32.876223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:32.876559 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:32.876489    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:32.876650 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:32.876557    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:33.876613 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:33.876580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:33.877045 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:33.876708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:34.876763 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:34.876723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:34.877221 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:34.876773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:34.877221 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:34.876855    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sdwt8" podUID="53076ba4-5cba-44f1-bf24-28f762dd893d"
Sep 18 21:51:34.877221 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:34.876986    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6bl8g" podUID="6fe437da-529d-46d9-a517-1d0b0ca1cff1"
Sep 18 21:51:35.656369 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:35.656340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:35.656642 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:35.656406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:35.656642 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.656518    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:35.656642 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.656542    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:35.656642 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.656595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:52:07.656574138 +0000 UTC m=+67.855282416 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:35.656642 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.656614    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret podName:53076ba4-5cba-44f1-bf24-28f762dd893d nodeName:}" failed. No retries permitted until 2026-09-18 21:52:07.656606032 +0000 UTC m=+67.855314311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret") pod "global-pull-secret-syncer-sdwt8" (UID: "53076ba4-5cba-44f1-bf24-28f762dd893d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:35.757777 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:35.757738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:35.757953 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.757884    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:35.757953 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.757905    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:35.757953 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.757918    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mbkn6 for pod openshift-network-diagnostics/network-check-target-6bl8g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:35.758069 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.757966    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6 podName:6fe437da-529d-46d9-a517-1d0b0ca1cff1 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:07.757953149 +0000 UTC m=+67.956661427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbkn6" (UniqueName: "kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6") pod "network-check-target-6bl8g" (UID: "6fe437da-529d-46d9-a517-1d0b0ca1cff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:35.876568 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:35.876536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:35.876739 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:35.876674    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wpmw9" podUID="5b87c66c-0ef7-4256-9c27-fe42e7d3edba"
Sep 18 21:51:36.316950 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.316920    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-13.ec2.internal" event="NodeReady"
Sep 18 21:51:36.317367 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.317043    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 21:51:36.367301 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.367277    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2l79"]
Sep 18 21:51:36.384944 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.384922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h8xj8"]
Sep 18 21:51:36.385048 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.385038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.387151 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.387128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:51:36.387151 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.387137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 21:51:36.387419 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.387190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 21:51:36.404895 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.404876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2l79"]
Sep 18 21:51:36.404990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.404915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h8xj8"]
Sep 18 21:51:36.405052 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.404999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.407351 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.407331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 21:51:36.407598 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.407582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 21:51:36.407709 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.407634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 21:51:36.408223 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.408182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vc2r9\""
Sep 18 21:51:36.564141 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee153d0-4d3a-432a-a011-cab546f03d5d-tmp-dir\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.564282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zc2\" (UniqueName: \"kubernetes.io/projected/9ee153d0-4d3a-432a-a011-cab546f03d5d-kube-api-access-42zc2\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.564282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czr2p\" (UniqueName: \"kubernetes.io/projected/cfeec5cb-1610-4b74-886b-d9c3740c0383-kube-api-access-czr2p\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.564282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfeec5cb-1610-4b74-886b-d9c3740c0383-cert\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.564282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee153d0-4d3a-432a-a011-cab546f03d5d-config-volume\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.564282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.564278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665297 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfeec5cb-1610-4b74-886b-d9c3740c0383-cert\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.665456 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee153d0-4d3a-432a-a011-cab546f03d5d-config-volume\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665456 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665456 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:36.665435    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:36.665622 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:36.665500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:37.165486346 +0000 UTC m=+37.364194625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:36.665622 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee153d0-4d3a-432a-a011-cab546f03d5d-tmp-dir\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665622 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zc2\" (UniqueName: \"kubernetes.io/projected/9ee153d0-4d3a-432a-a011-cab546f03d5d-kube-api-access-42zc2\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665622 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czr2p\" (UniqueName: \"kubernetes.io/projected/cfeec5cb-1610-4b74-886b-d9c3740c0383-kube-api-access-czr2p\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.665876 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee153d0-4d3a-432a-a011-cab546f03d5d-tmp-dir\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.665918 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.665878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee153d0-4d3a-432a-a011-cab546f03d5d-config-volume\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.669425 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.669387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfeec5cb-1610-4b74-886b-d9c3740c0383-cert\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.677595 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.677573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czr2p\" (UniqueName: \"kubernetes.io/projected/cfeec5cb-1610-4b74-886b-d9c3740c0383-kube-api-access-czr2p\") pod \"ingress-canary-h8xj8\" (UID: \"cfeec5cb-1610-4b74-886b-d9c3740c0383\") " pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.677716 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.677694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zc2\" (UniqueName: \"kubernetes.io/projected/9ee153d0-4d3a-432a-a011-cab546f03d5d-kube-api-access-42zc2\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:36.716114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.716096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h8xj8"
Sep 18 21:51:36.876348 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.876191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:51:36.876505 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.876235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:51:36.878985 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.878967    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:51:36.879127 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.879108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:51:36.879616 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.879600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:51:36.879713 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:36.879601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:51:37.048111 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.048082    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="e2b8d578abfcedcf7c676bb8ffe74c2aa2cb5363d45c6f518d9450eb2a28d135" exitCode=0
Sep 18 21:51:37.048265 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.048123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"e2b8d578abfcedcf7c676bb8ffe74c2aa2cb5363d45c6f518d9450eb2a28d135"}
Sep 18 21:51:37.052571 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.052545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h8xj8"]
Sep 18 21:51:37.056640 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:37.056616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeec5cb_1610_4b74_886b_d9c3740c0383.slice/crio-65fc2c1b37dffebe2d23b5fa667276526de34f2e72b58314551b9812bf33f39c WatchSource:0}: Error finding container 65fc2c1b37dffebe2d23b5fa667276526de34f2e72b58314551b9812bf33f39c: Status 404 returned error can't find the container with id 65fc2c1b37dffebe2d23b5fa667276526de34f2e72b58314551b9812bf33f39c
Sep 18 21:51:37.084232 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.084015    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"]
Sep 18 21:51:37.109954 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.109925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"]
Sep 18 21:51:37.109954 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.109935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.112606 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.112576    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 21:51:37.113074 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.112925    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 21:51:37.113074 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.112925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 21:51:37.113281 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.113265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 21:51:37.172038 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.171832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:37.172038 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:37.171940    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:37.172038 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:37.172002    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:38.171979848 +0000 UTC m=+38.370688130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:37.194559 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.194534    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"]
Sep 18 21:51:37.219945 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.219923    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"]
Sep 18 21:51:37.220027 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.220017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.223938 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.223921    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 21:51:37.224184 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.224170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 21:51:37.224956 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.224939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 21:51:37.224956 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.224950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 21:51:37.272571 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.272545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7zvv\" (UniqueName: \"kubernetes.io/projected/19c9ccd2-02a1-4123-947e-dd3b159697fe-kube-api-access-d7zvv\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.272680 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.272579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c9ccd2-02a1-4123-947e-dd3b159697fe-tmp\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.272736 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.272681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19c9ccd2-02a1-4123-947e-dd3b159697fe-klusterlet-config\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.372988 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.372964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19c9ccd2-02a1-4123-947e-dd3b159697fe-klusterlet-config\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.372992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92e686c-595f-4759-9936-230e52edde53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwr9\" (UniqueName: \"kubernetes.io/projected/b92e686c-595f-4759-9936-230e52edde53-kube-api-access-fnwr9\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7zvv\" (UniqueName: \"kubernetes.io/projected/19c9ccd2-02a1-4123-947e-dd3b159697fe-kube-api-access-d7zvv\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c9ccd2-02a1-4123-947e-dd3b159697fe-tmp\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.373625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.373540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19c9ccd2-02a1-4123-947e-dd3b159697fe-tmp\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.375895 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.375879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19c9ccd2-02a1-4123-947e-dd3b159697fe-klusterlet-config\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.382240 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.382220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7zvv\" (UniqueName: \"kubernetes.io/projected/19c9ccd2-02a1-4123-947e-dd3b159697fe-kube-api-access-d7zvv\") pod \"klusterlet-addon-workmgr-69b5d557d-fkt6h\" (UID: \"19c9ccd2-02a1-4123-947e-dd3b159697fe\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.422213 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.422191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:37.474116 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.474227 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92e686c-595f-4759-9936-230e52edde53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.474227 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwr9\" (UniqueName: \"kubernetes.io/projected/b92e686c-595f-4759-9936-230e52edde53-kube-api-access-fnwr9\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.474227 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.474353 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.474353 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.474301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.475058 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.475027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92e686c-595f-4759-9936-230e52edde53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.476790 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.476762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.476790 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.476773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-ca\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.477043 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.477014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-hub\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.477427 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.477390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b92e686c-595f-4759-9936-230e52edde53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.488001 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.487976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwr9\" (UniqueName: \"kubernetes.io/projected/b92e686c-595f-4759-9936-230e52edde53-kube-api-access-fnwr9\") pod \"cluster-proxy-proxy-agent-54f98588bb-ll4zm\" (UID: \"b92e686c-595f-4759-9936-230e52edde53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.540545 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.540514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:51:37.712736 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.712667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"]
Sep 18 21:51:37.715995 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:37.715965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c9ccd2_02a1_4123_947e_dd3b159697fe.slice/crio-683626a108c053b0afc34651a1d9399de2d89d74937698d763de82824f1655a6 WatchSource:0}: Error finding container 683626a108c053b0afc34651a1d9399de2d89d74937698d763de82824f1655a6: Status 404 returned error can't find the container with id 683626a108c053b0afc34651a1d9399de2d89d74937698d763de82824f1655a6
Sep 18 21:51:37.843111 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.843083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"]
Sep 18 21:51:37.846602 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:51:37.846571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb92e686c_595f_4759_9936_230e52edde53.slice/crio-31c6758423910b2236036d8201584f334f1217128bcd56eca3f5d8fa2c1a6f7c WatchSource:0}: Error finding container 31c6758423910b2236036d8201584f334f1217128bcd56eca3f5d8fa2c1a6f7c: Status 404 returned error can't find the container with id 31c6758423910b2236036d8201584f334f1217128bcd56eca3f5d8fa2c1a6f7c
Sep 18 21:51:37.876324 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.876299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:51:37.879917 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.879898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:51:37.880029 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:37.879965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:51:38.053069 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.052985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h8xj8" event={"ID":"cfeec5cb-1610-4b74-886b-d9c3740c0383","Type":"ContainerStarted","Data":"65fc2c1b37dffebe2d23b5fa667276526de34f2e72b58314551b9812bf33f39c"}
Sep 18 21:51:38.056608 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.056584    2563 generic.go:356] "Generic (PLEG): container finished" podID="625462d5-1360-4c58-9b03-2aee7bdebf98" containerID="8c7660cf3f3bb6c3631b2ca199aeb135ca59727da6831b02f35d00945885a382" exitCode=0
Sep 18 21:51:38.056732 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.056673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerDied","Data":"8c7660cf3f3bb6c3631b2ca199aeb135ca59727da6831b02f35d00945885a382"}
Sep 18 21:51:38.058000 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.057974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerStarted","Data":"31c6758423910b2236036d8201584f334f1217128bcd56eca3f5d8fa2c1a6f7c"}
Sep 18 21:51:38.059228 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.059208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h" event={"ID":"19c9ccd2-02a1-4123-947e-dd3b159697fe","Type":"ContainerStarted","Data":"683626a108c053b0afc34651a1d9399de2d89d74937698d763de82824f1655a6"}
Sep 18 21:51:38.179526 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:38.179499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:38.179755 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:38.179733    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:38.179866 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:38.179800    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:40.179781712 +0000 UTC m=+40.378489989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:39.067221 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:39.067141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" event={"ID":"625462d5-1360-4c58-9b03-2aee7bdebf98","Type":"ContainerStarted","Data":"927fb77d4c83f71f6680b45b6b6b56e7340eda5cd2426ff471cda00137ffeeac"}
Sep 18 21:51:39.092444 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:39.092373    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m2pkz" podStartSLOduration=9.829071891 podStartE2EDuration="38.092354072s" totalImagesPullingTime="28.263282181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:39.091446454 +0000 UTC m=+39.290154754" watchObservedRunningTime="2026-09-18 21:51:39.092354072 +0000 UTC m=+39.291062374"
Sep 18 21:51:40.197214 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:40.197167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:40.197813 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:40.197793    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:40.197910 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:40.197861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:44.197842905 +0000 UTC m=+44.396551184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:44.084112 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.083916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h8xj8" event={"ID":"cfeec5cb-1610-4b74-886b-d9c3740c0383","Type":"ContainerStarted","Data":"7e377418fc3c344b56a74282daa1097d17d9fd563b2aca1baf19af746e34ac03"}
Sep 18 21:51:44.086166 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.086141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerStarted","Data":"1406992ea0ad89ef14bd3bf9e95949328a9a02fdb6c65b4f92360a75b087452f"}
Sep 18 21:51:44.086272 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.086179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerStarted","Data":"1a3b5df6de1ce4d103fccd4b99f2d0fb4feda2bcb2acf77d45a02a444595d162"}
Sep 18 21:51:44.086272 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.086192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerStarted","Data":"2befe4266e4ac99a5ef802d7dd2b3ac3e67482a43ca5304678673c014f4157e4"}
Sep 18 21:51:44.087598 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.087576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h" event={"ID":"19c9ccd2-02a1-4123-947e-dd3b159697fe","Type":"ContainerStarted","Data":"0b7a6b4aecec2151742f7eb3dafdcccf2f3343d7e1eb682e48c2f6a7141c1a0d"}
Sep 18 21:51:44.087824 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.087805    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:44.089503 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.089485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h"
Sep 18 21:51:44.106872 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.106831    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h8xj8" podStartSLOduration=1.435947143 podStartE2EDuration="8.106818986s" totalImagesPullingTime="6.670871843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:44.106574019 +0000 UTC m=+44.305282330" watchObservedRunningTime="2026-09-18 21:51:44.106818986 +0000 UTC m=+44.305527284"
Sep 18 21:51:44.156179 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.156136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69b5d557d-fkt6h" podStartSLOduration=1.124905511 podStartE2EDuration="7.156119792s" totalImagesPullingTime="6.031214281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:44.155438518 +0000 UTC m=+44.354146814" watchObservedRunningTime="2026-09-18 21:51:44.156119792 +0000 UTC m=+44.354828092"
Sep 18 21:51:44.156862 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.156828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" podStartSLOduration=1.25938517 podStartE2EDuration="7.156819243s" totalImagesPullingTime="5.897434073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:44.12843454 +0000 UTC m=+44.327142839" watchObservedRunningTime="2026-09-18 21:51:44.156819243 +0000 UTC m=+44.355527543"
Sep 18 21:51:44.229029 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:44.229002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:44.229181 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:44.229162    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:44.229247 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:44.229236    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:52.229215291 +0000 UTC m=+52.427923587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:52.279871 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:51:52.279837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:51:52.280275 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:52.279975    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:52.280275 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:51:52.280044    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:52:08.280029927 +0000 UTC m=+68.478738204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:52:01.047801 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:01.047772    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvlk7"
Sep 18 21:52:07.691086 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.691051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:52:07.691086 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.691092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:52:07.697726 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.697707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:52:07.697786 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.697742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:52:07.701812 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:07.701796    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 21:52:07.701887 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:07.701877    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs podName:5b87c66c-0ef7-4256-9c27-fe42e7d3edba nodeName:}" failed. No retries permitted until 2026-09-18 21:53:11.701862809 +0000 UTC m=+131.900571085 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs") pod "network-metrics-daemon-wpmw9" (UID: "5b87c66c-0ef7-4256-9c27-fe42e7d3edba") : secret "metrics-daemon-secret" not found
Sep 18 21:52:07.704664 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.704645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53076ba4-5cba-44f1-bf24-28f762dd893d-original-pull-secret\") pod \"global-pull-secret-syncer-sdwt8\" (UID: \"53076ba4-5cba-44f1-bf24-28f762dd893d\") " pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:52:07.786525 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.786487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sdwt8"
Sep 18 21:52:07.791997 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.791977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:52:07.795968 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.795947    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:52:07.806140 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.806119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:52:07.815893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:07.815872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkn6\" (UniqueName: \"kubernetes.io/projected/6fe437da-529d-46d9-a517-1d0b0ca1cff1-kube-api-access-mbkn6\") pod \"network-check-target-6bl8g\" (UID: \"6fe437da-529d-46d9-a517-1d0b0ca1cff1\") " pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:52:08.062165 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.062076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sdwt8"]
Sep 18 21:52:08.066233 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:52:08.066207    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53076ba4_5cba_44f1_bf24_28f762dd893d.slice/crio-60d9a72d2c8aa9e1c2e3eb0d2aded0cedec6d24c22e3270c9517a243429d59e0 WatchSource:0}: Error finding container 60d9a72d2c8aa9e1c2e3eb0d2aded0cedec6d24c22e3270c9517a243429d59e0: Status 404 returned error can't find the container with id 60d9a72d2c8aa9e1c2e3eb0d2aded0cedec6d24c22e3270c9517a243429d59e0
Sep 18 21:52:08.094137 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.094109    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:52:08.102668 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.102648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:52:08.146055 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.145987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sdwt8" event={"ID":"53076ba4-5cba-44f1-bf24-28f762dd893d","Type":"ContainerStarted","Data":"60d9a72d2c8aa9e1c2e3eb0d2aded0cedec6d24c22e3270c9517a243429d59e0"}
Sep 18 21:52:08.296578 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.296528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:52:08.296731 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:08.296674    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:52:08.296771 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:08.296738    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:52:40.296720728 +0000 UTC m=+100.495429005 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:52:08.369940 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:08.369910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6bl8g"]
Sep 18 21:52:08.372866 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:52:08.372842    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe437da_529d_46d9_a517_1d0b0ca1cff1.slice/crio-eb7b055b9d718929f9e9b76fb60aa728398b877a7a32fd485ed07e11612c3c83 WatchSource:0}: Error finding container eb7b055b9d718929f9e9b76fb60aa728398b877a7a32fd485ed07e11612c3c83: Status 404 returned error can't find the container with id eb7b055b9d718929f9e9b76fb60aa728398b877a7a32fd485ed07e11612c3c83
Sep 18 21:52:09.150840 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:09.150799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bl8g" event={"ID":"6fe437da-529d-46d9-a517-1d0b0ca1cff1","Type":"ContainerStarted","Data":"eb7b055b9d718929f9e9b76fb60aa728398b877a7a32fd485ed07e11612c3c83"}
Sep 18 21:52:13.163655 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:13.163614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6bl8g" event={"ID":"6fe437da-529d-46d9-a517-1d0b0ca1cff1","Type":"ContainerStarted","Data":"2ca0eb3f0e9f1ff56ea36811c68a56781a0a30c9ff93ae8e47adea0de60ee696"}
Sep 18 21:52:13.164068 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:13.163721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:52:13.165252 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:13.165227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sdwt8" event={"ID":"53076ba4-5cba-44f1-bf24-28f762dd893d","Type":"ContainerStarted","Data":"18ab77b0e1e26fb8a487152695deeed87c78e45e22ae0392677d3a2ac8e5f375"}
Sep 18 21:52:13.207201 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:13.207157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sdwt8" podStartSLOduration=66.618289591 podStartE2EDuration="1m11.207141667s" totalImagesPullingTime="4.588852076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:13.206560874 +0000 UTC m=+73.405269174" watchObservedRunningTime="2026-09-18 21:52:13.207141667 +0000 UTC m=+73.405849967"
Sep 18 21:52:13.207811 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:13.207776    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6bl8g" podStartSLOduration=66.929145007 podStartE2EDuration="1m11.207765334s" totalImagesPullingTime="4.278620327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:13.184991565 +0000 UTC m=+73.383699864" watchObservedRunningTime="2026-09-18 21:52:13.207765334 +0000 UTC m=+73.406473694"
Sep 18 21:52:21.393750 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.393623    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xbczd"]
Sep 18 21:52:21.396197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.396177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.405472 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.405451    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 21:52:21.405573 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.405513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 21:52:21.406428 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.406385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 21:52:21.406428 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.406418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmpgg\""
Sep 18 21:52:21.406580 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.406408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 21:52:21.416084 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.416059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbczd"]
Sep 18 21:52:21.487543 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.487514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.487625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.487561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05b8f250-4be6-4182-9706-303d1f6ee7b9-crio-socket\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.487625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.487583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qb4z\" (UniqueName: \"kubernetes.io/projected/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-api-access-7qb4z\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.487693 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.487649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.487726 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.487697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05b8f250-4be6-4182-9706-303d1f6ee7b9-data-volume\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588693 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qb4z\" (UniqueName: \"kubernetes.io/projected/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-api-access-7qb4z\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05b8f250-4be6-4182-9706-303d1f6ee7b9-data-volume\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588861 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:21.588818    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:21.588861 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05b8f250-4be6-4182-9706-303d1f6ee7b9-crio-socket\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.588941 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:21.588889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.08886952 +0000 UTC m=+82.287577805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:21.588941 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:21.588820    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:21.588941 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.588910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05b8f250-4be6-4182-9706-303d1f6ee7b9-crio-socket\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.589070 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:21.588991    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.088969422 +0000 UTC m=+82.287677722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:21.589070 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.589058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05b8f250-4be6-4182-9706-303d1f6ee7b9-data-volume\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:21.597311 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:21.597284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qb4z\" (UniqueName: \"kubernetes.io/projected/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-api-access-7qb4z\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:22.092275 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:22.092239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:22.092470 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:22.092294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:22.092470 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:22.092365    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.092470 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:22.092434    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:23.092421451 +0000 UTC m=+83.291129728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.092470 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:22.092437    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:22.092650 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:22.092490    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:23.092475462 +0000 UTC m=+83.291183761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.099706 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:23.099672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:23.100076 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:23.099728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:23.100076 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:23.099791    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.100076 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:23.099814    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.100076 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:23.099863    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.099850202 +0000 UTC m=+85.298558480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.100076 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:23.099875    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.099869487 +0000 UTC m=+85.298577764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.498611 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:23.498532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9gnp_02a42efa-8572-4d8a-817a-f8d6542f0f4d/dns-node-resolver/0.log"
Sep 18 21:52:24.276095 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:24.276068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8w62_7f5e819a-8e56-41f7-ab2a-0d9f8a382813/node-ca/0.log"
Sep 18 21:52:24.675619 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:24.675591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h8xj8_cfeec5cb-1610-4b74-886b-d9c3740c0383/serve-healthcheck-canary/0.log"
Sep 18 21:52:25.114609 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:25.114575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:25.114772 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:25.114638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:25.114772 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:25.114732    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:25.114841 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:25.114802    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.114785565 +0000 UTC m=+89.313493841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:25.114841 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:25.114738    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:25.114919 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:25.114889    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.114875657 +0000 UTC m=+89.313583934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:29.139537 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:29.139501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:29.139929 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:29.139560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:29.139929 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:29.139629    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:29.139929 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:29.139646    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:29.139929 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:29.139729    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:37.139700137 +0000 UTC m=+97.338408414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:29.139929 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:29.139749    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:37.13973855 +0000 UTC m=+97.338446827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:37.190056 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:37.190029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:37.190427 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:37.190079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:37.190427 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:37.190172    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:37.190427 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:37.190230    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls podName:05b8f250-4be6-4182-9706-303d1f6ee7b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:53.190212717 +0000 UTC m=+113.388920996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xbczd" (UID: "05b8f250-4be6-4182-9706-303d1f6ee7b9") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:37.190567 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:37.190526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05b8f250-4be6-4182-9706-303d1f6ee7b9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:40.309244 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:40.309207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:52:40.309692 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:40.309328    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:52:40.309692 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:52:40.309381    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls podName:9ee153d0-4d3a-432a-a011-cab546f03d5d nodeName:}" failed. No retries permitted until 2026-09-18 21:53:44.309366528 +0000 UTC m=+164.508074808 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls") pod "dns-default-n2l79" (UID: "9ee153d0-4d3a-432a-a011-cab546f03d5d") : secret "dns-default-metrics-tls" not found
Sep 18 21:52:44.171872 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:44.171835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6bl8g"
Sep 18 21:52:53.191693 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:53.191659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:53.193913 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:53.193889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05b8f250-4be6-4182-9706-303d1f6ee7b9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbczd\" (UID: \"05b8f250-4be6-4182-9706-303d1f6ee7b9\") " pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:53.204875 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:53.204855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbczd"
Sep 18 21:52:53.483332 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:53.483311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbczd"]
Sep 18 21:52:53.485833 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:52:53.485807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b8f250_4be6_4182_9706_303d1f6ee7b9.slice/crio-34e0f218c1fedde89e85c055e3bd82ea3ec4406e4742293f318d0df84fe7837e WatchSource:0}: Error finding container 34e0f218c1fedde89e85c055e3bd82ea3ec4406e4742293f318d0df84fe7837e: Status 404 returned error can't find the container with id 34e0f218c1fedde89e85c055e3bd82ea3ec4406e4742293f318d0df84fe7837e
Sep 18 21:52:54.287282 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:54.287251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbczd" event={"ID":"05b8f250-4be6-4182-9706-303d1f6ee7b9","Type":"ContainerStarted","Data":"e8aed92f3531e28197b28c7e4c559893e2fd47a443b9f68ffcebcc633a4cead2"}
Sep 18 21:52:54.287622 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:54.287316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbczd" event={"ID":"05b8f250-4be6-4182-9706-303d1f6ee7b9","Type":"ContainerStarted","Data":"1fed0e88ba02c9c8c611a1d2a4a730988eb22cc377fb616e21a55f74d8c88574"}
Sep 18 21:52:54.287622 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:54.287345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbczd" event={"ID":"05b8f250-4be6-4182-9706-303d1f6ee7b9","Type":"ContainerStarted","Data":"34e0f218c1fedde89e85c055e3bd82ea3ec4406e4742293f318d0df84fe7837e"}
Sep 18 21:52:55.303874 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.303826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55ff9774c5-tss26"]
Sep 18 21:52:55.306007 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.305985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.310696 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.310668    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 21:52:55.311633 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.311574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xmfvd\""
Sep 18 21:52:55.311771 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.311655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 21:52:55.311893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.311867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 21:52:55.317296 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.317275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 21:52:55.339782 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.339755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55ff9774c5-tss26"]
Sep 18 21:52:55.406685 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-image-registry-private-configuration\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406786 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-tls\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406786 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67093347-f081-4c7f-b51f-1e31b4ef3b76-ca-trust-extracted\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406786 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-trusted-ca\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406902 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-bound-sa-token\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406902 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-installation-pull-secrets\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406902 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-certificates\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.406902 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.406899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngt8\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-kube-api-access-wngt8\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507679 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67093347-f081-4c7f-b51f-1e31b4ef3b76-ca-trust-extracted\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507781 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-trusted-ca\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507781 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-bound-sa-token\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507781 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-installation-pull-secrets\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507899 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-certificates\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507899 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngt8\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-kube-api-access-wngt8\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-image-registry-private-configuration\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.507990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.507965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-tls\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.508090 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.508039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67093347-f081-4c7f-b51f-1e31b4ef3b76-ca-trust-extracted\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.508634 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.508601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-trusted-ca\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.508740 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.508642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-certificates\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.510058 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.510043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-installation-pull-secrets\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.510590 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.510570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-registry-tls\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.510678 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.510658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67093347-f081-4c7f-b51f-1e31b4ef3b76-image-registry-private-configuration\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.525816 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.525799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngt8\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-kube-api-access-wngt8\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.529493 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.529472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67093347-f081-4c7f-b51f-1e31b4ef3b76-bound-sa-token\") pod \"image-registry-55ff9774c5-tss26\" (UID: \"67093347-f081-4c7f-b51f-1e31b4ef3b76\") " pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.618214 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.618192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:55.897770 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:55.897702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55ff9774c5-tss26"]
Sep 18 21:52:55.900550 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:52:55.900527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67093347_f081_4c7f_b51f_1e31b4ef3b76.slice/crio-68d169e091be25fc2770e339d13820ccec7f12e6d79b01e4df5fdc88d72423a6 WatchSource:0}: Error finding container 68d169e091be25fc2770e339d13820ccec7f12e6d79b01e4df5fdc88d72423a6: Status 404 returned error can't find the container with id 68d169e091be25fc2770e339d13820ccec7f12e6d79b01e4df5fdc88d72423a6
Sep 18 21:52:56.295547 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.295460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55ff9774c5-tss26" event={"ID":"67093347-f081-4c7f-b51f-1e31b4ef3b76","Type":"ContainerStarted","Data":"78e0a03b7c73c250b92de6676932abdd3c5492306f2d4c7890e700d66c1c1d25"}
Sep 18 21:52:56.295547 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.295509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55ff9774c5-tss26" event={"ID":"67093347-f081-4c7f-b51f-1e31b4ef3b76","Type":"ContainerStarted","Data":"68d169e091be25fc2770e339d13820ccec7f12e6d79b01e4df5fdc88d72423a6"}
Sep 18 21:52:56.295755 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.295564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:52:56.297508 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.297483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbczd" event={"ID":"05b8f250-4be6-4182-9706-303d1f6ee7b9","Type":"ContainerStarted","Data":"4029bf6f8b28559a08f4739ee7b2405621f174c3f28d4b76122db2163b9161a2"}
Sep 18 21:52:56.316892 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.316852    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55ff9774c5-tss26" podStartSLOduration=1.3168355809999999 podStartE2EDuration="1.316835581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:56.316755692 +0000 UTC m=+116.515463994" watchObservedRunningTime="2026-09-18 21:52:56.316835581 +0000 UTC m=+116.515543881"
Sep 18 21:52:56.339172 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:56.339135    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xbczd" podStartSLOduration=33.527404644 podStartE2EDuration="35.339123005s" totalImagesPullingTime="1.811718361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:52:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:56.337997323 +0000 UTC m=+116.536705623" watchObservedRunningTime="2026-09-18 21:52:56.339123005 +0000 UTC m=+116.537831303"
Sep 18 21:52:57.542447 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:52:57.542386    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" podUID="b92e686c-595f-4759-9936-230e52edde53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 21:53:03.510949 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.510919    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfl9r"]
Sep 18 21:53:03.513944 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.513927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.518232 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.518210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 21:53:03.518678 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.518653    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 21:53:03.518782 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.518726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mpctd\""
Sep 18 21:53:03.518844 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.518833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 21:53:03.518963 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.518946    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 21:53:03.519026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.519013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 21:53:03.519175 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.519156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 21:53:03.659685 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-textfile\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659797 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-root\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659797 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659864 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-wtmp\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659864 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x54x\" (UniqueName: \"kubernetes.io/projected/781e40d8-b59e-40bb-9de7-8366751c58a1-kube-api-access-2x54x\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659925 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-sys\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659925 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-metrics-client-ca\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659987 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.659987 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.659951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-tls\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760761 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-textfile\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760849 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-root\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760849 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-wtmp\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x54x\" (UniqueName: \"kubernetes.io/projected/781e40d8-b59e-40bb-9de7-8366751c58a1-kube-api-access-2x54x\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-sys\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-root\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.760976 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.760937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-metrics-client-ca\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761268 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761268 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-tls\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761268 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-textfile\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761268 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-sys\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761473 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-wtmp\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761530 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-metrics-client-ca\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.761615 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.761593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.763347 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.763321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-tls\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.763347 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.763343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/781e40d8-b59e-40bb-9de7-8366751c58a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.771920 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.771894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x54x\" (UniqueName: \"kubernetes.io/projected/781e40d8-b59e-40bb-9de7-8366751c58a1-kube-api-access-2x54x\") pod \"node-exporter-pfl9r\" (UID: \"781e40d8-b59e-40bb-9de7-8366751c58a1\") " pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.823118 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:03.823096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfl9r"
Sep 18 21:53:03.831249 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:53:03.831223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781e40d8_b59e_40bb_9de7_8366751c58a1.slice/crio-4bcce5fd3b53e448cd67ed4464e88ee89705cef5df315a9a0424ca8d69f49c5f WatchSource:0}: Error finding container 4bcce5fd3b53e448cd67ed4464e88ee89705cef5df315a9a0424ca8d69f49c5f: Status 404 returned error can't find the container with id 4bcce5fd3b53e448cd67ed4464e88ee89705cef5df315a9a0424ca8d69f49c5f
Sep 18 21:53:04.325117 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:04.325076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfl9r" event={"ID":"781e40d8-b59e-40bb-9de7-8366751c58a1","Type":"ContainerStarted","Data":"4bcce5fd3b53e448cd67ed4464e88ee89705cef5df315a9a0424ca8d69f49c5f"}
Sep 18 21:53:05.329294 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:05.329260    2563 generic.go:356] "Generic (PLEG): container finished" podID="781e40d8-b59e-40bb-9de7-8366751c58a1" containerID="5c4ab7190c562387e93ce2d2cf129dba517897a6b313e26f5771f0f8ea650211" exitCode=0
Sep 18 21:53:05.329657 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:05.329353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfl9r" event={"ID":"781e40d8-b59e-40bb-9de7-8366751c58a1","Type":"ContainerDied","Data":"5c4ab7190c562387e93ce2d2cf129dba517897a6b313e26f5771f0f8ea650211"}
Sep 18 21:53:06.333875 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:06.333846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfl9r" event={"ID":"781e40d8-b59e-40bb-9de7-8366751c58a1","Type":"ContainerStarted","Data":"36078eec10fd1ec7e730eda89859c49a3cb35dcc2a49c72575e9f6f01c99b0be"}
Sep 18 21:53:06.334232 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:06.333891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfl9r" event={"ID":"781e40d8-b59e-40bb-9de7-8366751c58a1","Type":"ContainerStarted","Data":"bd424b90c95403e0bb6cedbe5c092030c37a368f87a026c4160cdc1da7f3d6f7"}
Sep 18 21:53:06.355252 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:06.355208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfl9r" podStartSLOduration=2.430643786 podStartE2EDuration="3.355194025s" totalImagesPullingTime="863.960652ms" totalInitContainerRuntime="60.589587ms" isStatefulPod=true podCreationTimestamp="2026-09-18 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:06.354423151 +0000 UTC m=+126.553131447" watchObservedRunningTime="2026-09-18 21:53:06.355194025 +0000 UTC m=+126.553902701"
Sep 18 21:53:07.542011 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:07.541965    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" podUID="b92e686c-595f-4759-9936-230e52edde53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 21:53:11.716411 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:11.716371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:53:11.718798 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:11.718772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b87c66c-0ef7-4256-9c27-fe42e7d3edba-metrics-certs\") pod \"network-metrics-daemon-wpmw9\" (UID: \"5b87c66c-0ef7-4256-9c27-fe42e7d3edba\") " pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:53:11.790498 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:11.790470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:53:11.797110 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:11.797091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wpmw9"
Sep 18 21:53:12.095343 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:12.095309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wpmw9"]
Sep 18 21:53:12.098658 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:53:12.098632    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b87c66c_0ef7_4256_9c27_fe42e7d3edba.slice/crio-ad20a52df903e660250a23dae6378afa294b1b77d729b1e084607c27edfc34a4 WatchSource:0}: Error finding container ad20a52df903e660250a23dae6378afa294b1b77d729b1e084607c27edfc34a4: Status 404 returned error can't find the container with id ad20a52df903e660250a23dae6378afa294b1b77d729b1e084607c27edfc34a4
Sep 18 21:53:12.359188 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:12.359112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpmw9" event={"ID":"5b87c66c-0ef7-4256-9c27-fe42e7d3edba","Type":"ContainerStarted","Data":"ad20a52df903e660250a23dae6378afa294b1b77d729b1e084607c27edfc34a4"}
Sep 18 21:53:13.364119 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:13.364085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpmw9" event={"ID":"5b87c66c-0ef7-4256-9c27-fe42e7d3edba","Type":"ContainerStarted","Data":"1b37d8e3999c4b099685bf58a0251c05f7999d553d916ef53b1c8814fb4d0109"}
Sep 18 21:53:13.364508 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:13.364149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wpmw9" event={"ID":"5b87c66c-0ef7-4256-9c27-fe42e7d3edba","Type":"ContainerStarted","Data":"5cc31e516fca0aa88339cbcfb9a41640e478c44fc9e6551619058da87fb83be2"}
Sep 18 21:53:13.391803 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:13.391761    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wpmw9" podStartSLOduration=130.535431201 podStartE2EDuration="2m11.391744865s" totalImagesPullingTime="856.313664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:13.390837153 +0000 UTC m=+133.589545442" watchObservedRunningTime="2026-09-18 21:53:13.391744865 +0000 UTC m=+133.590453163"
Sep 18 21:53:15.622716 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:15.622683    2563 patch_prober.go:28] interesting pod/image-registry-55ff9774c5-tss26 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 21:53:15.623077 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:15.622731    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55ff9774c5-tss26" podUID="67093347-f081-4c7f-b51f-1e31b4ef3b76" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 21:53:17.305047 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:17.305022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55ff9774c5-tss26"
Sep 18 21:53:17.541883 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:17.541850    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" podUID="b92e686c-595f-4759-9936-230e52edde53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 21:53:17.542032 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:17.541914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm"
Sep 18 21:53:17.542342 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:17.542315    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1406992ea0ad89ef14bd3bf9e95949328a9a02fdb6c65b4f92360a75b087452f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 18 21:53:17.542388 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:17.542364    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" podUID="b92e686c-595f-4759-9936-230e52edde53" containerName="service-proxy" containerID="cri-o://1406992ea0ad89ef14bd3bf9e95949328a9a02fdb6c65b4f92360a75b087452f" gracePeriod=30
Sep 18 21:53:18.385038 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:18.385008    2563 generic.go:356] "Generic (PLEG): container finished" podID="b92e686c-595f-4759-9936-230e52edde53" containerID="1406992ea0ad89ef14bd3bf9e95949328a9a02fdb6c65b4f92360a75b087452f" exitCode=2
Sep 18 21:53:18.385422 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:18.385051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerDied","Data":"1406992ea0ad89ef14bd3bf9e95949328a9a02fdb6c65b4f92360a75b087452f"}
Sep 18 21:53:18.385422 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:18.385083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54f98588bb-ll4zm" event={"ID":"b92e686c-595f-4759-9936-230e52edde53","Type":"ContainerStarted","Data":"97a6b5405a6fa11375ae75b29dfc70f576679a887c321ec7ac5f9ca74805c5e4"}
Sep 18 21:53:24.381762 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.381729    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:53:24.385872 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.385853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.388541 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.388522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389484    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 21:53:24.389566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nz2c\""
Sep 18 21:53:24.389830 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.389746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 21:53:24.397575 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.397556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:53:24.399218 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.399196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 21:53:24.506701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506808 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506808 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506808 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506911 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506911 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbbj\" (UniqueName: \"kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.506911 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.506886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607568 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607661 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607661 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607661 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607793 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607793 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.607793 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.607741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbbj\" (UniqueName: \"kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.608406 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.608363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.608537 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.608520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.608586 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.608565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.608731 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.608717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.610653 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.610632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.610866 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.610843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.616209 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.616189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbbj\" (UniqueName: \"kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj\") pod \"console-67d99fc6b6-z6269\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") " pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.695631 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.695573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:24.973272 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:24.973209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:53:24.976327 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:53:24.976302    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645bef73_891c_401a_b373_29ee9961cfce.slice/crio-d7805a416e80690a9f83e5f43e0f1321390ba0a3c7e04aabcbdfb8e4dd2a9a8c WatchSource:0}: Error finding container d7805a416e80690a9f83e5f43e0f1321390ba0a3c7e04aabcbdfb8e4dd2a9a8c: Status 404 returned error can't find the container with id d7805a416e80690a9f83e5f43e0f1321390ba0a3c7e04aabcbdfb8e4dd2a9a8c
Sep 18 21:53:25.413709 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:25.413630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d99fc6b6-z6269" event={"ID":"645bef73-891c-401a-b373-29ee9961cfce","Type":"ContainerStarted","Data":"d7805a416e80690a9f83e5f43e0f1321390ba0a3c7e04aabcbdfb8e4dd2a9a8c"}
Sep 18 21:53:29.429943 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:29.429909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d99fc6b6-z6269" event={"ID":"645bef73-891c-401a-b373-29ee9961cfce","Type":"ContainerStarted","Data":"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"}
Sep 18 21:53:29.453236 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:29.453184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d99fc6b6-z6269" podStartSLOduration=1.711499917 podStartE2EDuration="5.453170428s" totalImagesPullingTime="3.741670511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:29.452181843 +0000 UTC m=+149.650890155" watchObservedRunningTime="2026-09-18 21:53:29.453170428 +0000 UTC m=+149.651878727"
Sep 18 21:53:34.696556 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:34.696523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:34.696927 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:34.696600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:34.701203 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:34.701177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:35.452040 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:35.452008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:53:39.394364 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:53:39.394325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n2l79" podUID="9ee153d0-4d3a-432a-a011-cab546f03d5d"
Sep 18 21:53:39.460733 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:39.460711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2l79"
Sep 18 21:53:44.343685 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:44.343653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:53:44.345884 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:44.345865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ee153d0-4d3a-432a-a011-cab546f03d5d-metrics-tls\") pod \"dns-default-n2l79\" (UID: \"9ee153d0-4d3a-432a-a011-cab546f03d5d\") " pod="openshift-dns/dns-default-n2l79"
Sep 18 21:53:44.564100 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:44.564068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:53:44.571556 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:44.571536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2l79"
Sep 18 21:53:44.853929 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:44.853900    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2l79"]
Sep 18 21:53:44.857061 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:53:44.857033    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee153d0_4d3a_432a_a011_cab546f03d5d.slice/crio-efe4c5566b7a4db3d35c23bca8fbe9028e4b8e0e539b27454555ee1695ed2bd9 WatchSource:0}: Error finding container efe4c5566b7a4db3d35c23bca8fbe9028e4b8e0e539b27454555ee1695ed2bd9: Status 404 returned error can't find the container with id efe4c5566b7a4db3d35c23bca8fbe9028e4b8e0e539b27454555ee1695ed2bd9
Sep 18 21:53:45.483011 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:45.482974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2l79" event={"ID":"9ee153d0-4d3a-432a-a011-cab546f03d5d","Type":"ContainerStarted","Data":"efe4c5566b7a4db3d35c23bca8fbe9028e4b8e0e539b27454555ee1695ed2bd9"}
Sep 18 21:53:46.495166 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:46.495132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2l79" event={"ID":"9ee153d0-4d3a-432a-a011-cab546f03d5d","Type":"ContainerStarted","Data":"71b59ebea4db2ebd834348eda3e3ef6b87d7dae6e924473d2fd941cdde290ce6"}
Sep 18 21:53:47.500718 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:47.500684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2l79" event={"ID":"9ee153d0-4d3a-432a-a011-cab546f03d5d","Type":"ContainerStarted","Data":"9237a569e2fce57709a44dee2cee481c0403f4448adfe3cb7ec829ec62266383"}
Sep 18 21:53:47.501134 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:47.500817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2l79"
Sep 18 21:53:47.523082 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:47.523036    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n2l79" podStartSLOduration=130.019850986 podStartE2EDuration="2m11.523023718s" totalImagesPullingTime="1.503172732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:47.521948187 +0000 UTC m=+167.720656487" watchObservedRunningTime="2026-09-18 21:53:47.523023718 +0000 UTC m=+167.721732016"
Sep 18 21:53:57.505913 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:53:57.505879    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2l79"
Sep 18 21:54:30.710778 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.710746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:54:30.714044 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.714028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.734088 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.734063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:54:30.840083 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840189 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840189 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840270 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840270 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840270 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.840373 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.840271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlb55\" (UniqueName: \"kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941514 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941590 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941590 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941811 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941862 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941908 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.941908 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.941888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlb55\" (UniqueName: \"kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.942437 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.942414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.942549 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.942449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.942549 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.942480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.942678 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.942600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.943995 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.943970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.944102 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.944084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:30.951112 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:30.951090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlb55\" (UniqueName: \"kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55\") pod \"console-57995b59b9-47wqx\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") " pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:31.022236 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:31.022186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:31.301703 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:31.301641    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:54:31.304368 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:54:31.304342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4818bf_acf9_45df_9eac_2ea997fed18c.slice/crio-e2cb7e19c844cae82fa431d34665bde6b2b5e420749f843ca86c77a943a72615 WatchSource:0}: Error finding container e2cb7e19c844cae82fa431d34665bde6b2b5e420749f843ca86c77a943a72615: Status 404 returned error can't find the container with id e2cb7e19c844cae82fa431d34665bde6b2b5e420749f843ca86c77a943a72615
Sep 18 21:54:31.652389 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:31.652354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57995b59b9-47wqx" event={"ID":"7d4818bf-acf9-45df-9eac-2ea997fed18c","Type":"ContainerStarted","Data":"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"}
Sep 18 21:54:31.652553 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:31.652445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57995b59b9-47wqx" event={"ID":"7d4818bf-acf9-45df-9eac-2ea997fed18c","Type":"ContainerStarted","Data":"e2cb7e19c844cae82fa431d34665bde6b2b5e420749f843ca86c77a943a72615"}
Sep 18 21:54:31.692706 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:31.692662    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57995b59b9-47wqx" podStartSLOduration=1.692647191 podStartE2EDuration="1.692647191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:54:31.6919131 +0000 UTC m=+211.890621400" watchObservedRunningTime="2026-09-18 21:54:31.692647191 +0000 UTC m=+211.891355549"
Sep 18 21:54:41.022427 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:41.022375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:41.022965 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:41.022449    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:41.027292 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:41.027268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:41.690548 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:41.690518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:54:41.747790 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:54:41.747758    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:55:06.766283 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:06.766229    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67d99fc6b6-z6269" podUID="645bef73-891c-401a-b373-29ee9961cfce" containerName="console" containerID="cri-o://bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56" gracePeriod=15
Sep 18 21:55:06.999454 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:06.999432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:55:07.081717 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfbbj\" (UniqueName: \"kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.081717 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.081890 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.081890 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.081890 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.082037 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.082037 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.081938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca\") pod \"645bef73-891c-401a-b373-29ee9961cfce\" (UID: \"645bef73-891c-401a-b373-29ee9961cfce\") "
Sep 18 21:55:07.082233 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.082209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:55:07.082305 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.082232    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:55:07.082364 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.082261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config" (OuterVolumeSpecName: "console-config") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:55:07.082462 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.082437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca" (OuterVolumeSpecName: "service-ca") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:55:07.083870 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.083843    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:55:07.084039 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.084026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj" (OuterVolumeSpecName: "kube-api-access-wfbbj") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "kube-api-access-wfbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:55:07.084179 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.084154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "645bef73-891c-401a-b373-29ee9961cfce" (UID: "645bef73-891c-401a-b373-29ee9961cfce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:55:07.182566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbbj\" (UniqueName: \"kubernetes.io/projected/645bef73-891c-401a-b373-29ee9961cfce-kube-api-access-wfbbj\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182566 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182565    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-oauth-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182687 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182578    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-trusted-ca-bundle\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182687 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182591    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-oauth-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182687 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182599    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/645bef73-891c-401a-b373-29ee9961cfce-console-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182687 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182608    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-console-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.182687 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.182616    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/645bef73-891c-401a-b373-29ee9961cfce-service-ca\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:55:07.778873 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.778839    2563 generic.go:356] "Generic (PLEG): container finished" podID="645bef73-891c-401a-b373-29ee9961cfce" containerID="bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56" exitCode=0
Sep 18 21:55:07.779291 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.778886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d99fc6b6-z6269"
Sep 18 21:55:07.779291 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.778921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d99fc6b6-z6269" event={"ID":"645bef73-891c-401a-b373-29ee9961cfce","Type":"ContainerDied","Data":"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"}
Sep 18 21:55:07.779291 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.778958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d99fc6b6-z6269" event={"ID":"645bef73-891c-401a-b373-29ee9961cfce","Type":"ContainerDied","Data":"d7805a416e80690a9f83e5f43e0f1321390ba0a3c7e04aabcbdfb8e4dd2a9a8c"}
Sep 18 21:55:07.779291 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.778986    2563 scope.go:117] "RemoveContainer" containerID="bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"
Sep 18 21:55:07.787932 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.787918    2563 scope.go:117] "RemoveContainer" containerID="bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"
Sep 18 21:55:07.788205 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:55:07.788186    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56\": container with ID starting with bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56 not found: ID does not exist" containerID="bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"
Sep 18 21:55:07.788271 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.788213    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56"} err="failed to get container status \"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56\": rpc error: code = NotFound desc = could not find container \"bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56\": container with ID starting with bc49446e5e58bd9c5717706809cd115f66142f9e4aae72dab8519d3ddda6fb56 not found: ID does not exist"
Sep 18 21:55:07.801662 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.801638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:55:07.807542 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.807522    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d99fc6b6-z6269"]
Sep 18 21:55:07.880264 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:07.880245    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645bef73-891c-401a-b373-29ee9961cfce" path="/var/lib/kubelet/pods/645bef73-891c-401a-b373-29ee9961cfce/volumes"
Sep 18 21:55:48.733684 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.733652    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:55:48.734123 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.733878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645bef73-891c-401a-b373-29ee9961cfce" containerName="console"
Sep 18 21:55:48.734123 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.733888    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="645bef73-891c-401a-b373-29ee9961cfce" containerName="console"
Sep 18 21:55:48.734123 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.733937    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="645bef73-891c-401a-b373-29ee9961cfce" containerName="console"
Sep 18 21:55:48.736522 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.736506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.750271 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.750248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:55:48.850660 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850766 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850944 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmkg\" (UniqueName: \"kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.850944 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.850803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951312 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmkg\" (UniqueName: \"kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.951703 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.951470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.952126 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.952101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.952258 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.952233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.952324 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.952287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.952614 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.952588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.953927 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.953896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.954069 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.954048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:48.959593 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:48.959573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmkg\" (UniqueName: \"kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg\") pod \"console-676985698c-5pvlz\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") " pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:49.045221 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:49.045161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:49.318574 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:49.318439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:55:49.321054 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:55:49.321019    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a605bff_bac1_47cd_87d8_fcb2668e174d.slice/crio-70b31eb7c8ed48b29ccbad9cfb5610cc49e98b32539720a81f933b2b113b3536 WatchSource:0}: Error finding container 70b31eb7c8ed48b29ccbad9cfb5610cc49e98b32539720a81f933b2b113b3536: Status 404 returned error can't find the container with id 70b31eb7c8ed48b29ccbad9cfb5610cc49e98b32539720a81f933b2b113b3536
Sep 18 21:55:49.926490 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:49.926448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676985698c-5pvlz" event={"ID":"3a605bff-bac1-47cd-87d8-fcb2668e174d","Type":"ContainerStarted","Data":"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"}
Sep 18 21:55:49.926863 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:49.926508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676985698c-5pvlz" event={"ID":"3a605bff-bac1-47cd-87d8-fcb2668e174d","Type":"ContainerStarted","Data":"70b31eb7c8ed48b29ccbad9cfb5610cc49e98b32539720a81f933b2b113b3536"}
Sep 18 21:55:49.949603 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:49.949557    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-676985698c-5pvlz" podStartSLOduration=1.949542648 podStartE2EDuration="1.949542648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:55:49.948476765 +0000 UTC m=+290.147185064" watchObservedRunningTime="2026-09-18 21:55:49.949542648 +0000 UTC m=+290.148250946"
Sep 18 21:55:59.045812 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:59.045782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:59.046165 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:59.045835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:59.050342 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:59.050321    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:55:59.963357 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:55:59.963329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:56:00.042563 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:00.042534    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:56:01.805356 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:01.805331    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:56:01.807334 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:01.807312    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 21:56:01.812757 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:01.812736    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 21:56:25.061235 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.061194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57995b59b9-47wqx" podUID="7d4818bf-acf9-45df-9eac-2ea997fed18c" containerName="console" containerID="cri-o://1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353" gracePeriod=15
Sep 18 21:56:25.298094 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.298072    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:56:25.377108 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlb55\" (UniqueName: \"kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377108 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377179    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377240    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle\") pod \"7d4818bf-acf9-45df-9eac-2ea997fed18c\" (UID: \"7d4818bf-acf9-45df-9eac-2ea997fed18c\") "
Sep 18 21:56:25.377692 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config" (OuterVolumeSpecName: "console-config") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:56:25.377750 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:56:25.377793 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377762    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:56:25.377829 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.377799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:56:25.379212 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.379193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:56:25.379327 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.379306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55" (OuterVolumeSpecName: "kube-api-access-hlb55") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "kube-api-access-hlb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:56:25.379391 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.379331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d4818bf-acf9-45df-9eac-2ea997fed18c" (UID: "7d4818bf-acf9-45df-9eac-2ea997fed18c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:56:25.477771 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477747    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-oauth-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477771 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477768    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-service-ca\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477777    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-trusted-ca-bundle\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477785    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlb55\" (UniqueName: \"kubernetes.io/projected/7d4818bf-acf9-45df-9eac-2ea997fed18c-kube-api-access-hlb55\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477794    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-oauth-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477802    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:25.477893 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:25.477810    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d4818bf-acf9-45df-9eac-2ea997fed18c-console-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:56:26.043478 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.043444    2563 generic.go:356] "Generic (PLEG): container finished" podID="7d4818bf-acf9-45df-9eac-2ea997fed18c" containerID="1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353" exitCode=0
Sep 18 21:56:26.043635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.043485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57995b59b9-47wqx" event={"ID":"7d4818bf-acf9-45df-9eac-2ea997fed18c","Type":"ContainerDied","Data":"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"}
Sep 18 21:56:26.043635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.043488    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57995b59b9-47wqx"
Sep 18 21:56:26.043635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.043518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57995b59b9-47wqx" event={"ID":"7d4818bf-acf9-45df-9eac-2ea997fed18c","Type":"ContainerDied","Data":"e2cb7e19c844cae82fa431d34665bde6b2b5e420749f843ca86c77a943a72615"}
Sep 18 21:56:26.043635 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.043537    2563 scope.go:117] "RemoveContainer" containerID="1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"
Sep 18 21:56:26.051130 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.051114    2563 scope.go:117] "RemoveContainer" containerID="1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"
Sep 18 21:56:26.051429 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:56:26.051407    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353\": container with ID starting with 1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353 not found: ID does not exist" containerID="1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"
Sep 18 21:56:26.051484 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.051439    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353"} err="failed to get container status \"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353\": rpc error: code = NotFound desc = could not find container \"1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353\": container with ID starting with 1d85af59c0b63afa6e0b0d2389e76f0f2a2626d551c34e28871dcff692b45353 not found: ID does not exist"
Sep 18 21:56:26.070083 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.070059    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:56:26.074980 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:26.074961    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57995b59b9-47wqx"]
Sep 18 21:56:27.881053 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:56:27.881020    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4818bf-acf9-45df-9eac-2ea997fed18c" path="/var/lib/kubelet/pods/7d4818bf-acf9-45df-9eac-2ea997fed18c/volumes"
Sep 18 21:58:25.384658 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.384583    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67b6fb4769-2gf85"]
Sep 18 21:58:25.385079 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.384795    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4818bf-acf9-45df-9eac-2ea997fed18c" containerName="console"
Sep 18 21:58:25.385079 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.384807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4818bf-acf9-45df-9eac-2ea997fed18c" containerName="console"
Sep 18 21:58:25.385079 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.384853    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4818bf-acf9-45df-9eac-2ea997fed18c" containerName="console"
Sep 18 21:58:25.387531 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.387515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.397133 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:58:25.397100    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-67b6fb4769-2gf85\" is forbidden: User \"system:node:ip-10-0-141-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-141-13.ec2.internal' and this object" podUID="26cf3522-8185-49ea-bbee-4896e5224e64" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.449057 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.449030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b6fb4769-2gf85"]
Sep 18 21:58:25.572982 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.572945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-trusted-ca-bundle\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.572994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gpd\" (UniqueName: \"kubernetes.io/projected/26cf3522-8185-49ea-bbee-4896e5224e64-kube-api-access-s9gpd\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.573025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-oauth-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.573105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-oauth-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.573153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-service-ca\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573197 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.573196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.573434 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.573263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-console-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.574443 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.574419    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"]
Sep 18 21:58:25.577406 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.577380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.585764 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.585742    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 21:58:25.585900 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.585885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zl772\""
Sep 18 21:58:25.586220 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.586206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:58:25.605618 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.605593    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"]
Sep 18 21:58:25.673681 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-console-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673681 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-trusted-ca-bundle\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673808 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gpd\" (UniqueName: \"kubernetes.io/projected/26cf3522-8185-49ea-bbee-4896e5224e64-kube-api-access-s9gpd\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673808 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-oauth-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673906 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-oauth-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673906 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-service-ca\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.673906 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.673878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.674443 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.674419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-console-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.674547 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.674506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-service-ca\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.674603 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.674557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-trusted-ca-bundle\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.674640 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.674617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26cf3522-8185-49ea-bbee-4896e5224e64-oauth-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.676922 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.676893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-oauth-config\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.677000 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.676901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26cf3522-8185-49ea-bbee-4896e5224e64-console-serving-cert\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.684648 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.684629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gpd\" (UniqueName: \"kubernetes.io/projected/26cf3522-8185-49ea-bbee-4896e5224e64-kube-api-access-s9gpd\") pod \"console-67b6fb4769-2gf85\" (UID: \"26cf3522-8185-49ea-bbee-4896e5224e64\") " pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.695445 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.695426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:25.775059 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.775033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92466bbf-231a-4554-83e9-136ee0a63a95-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.775174 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.775103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztgx\" (UniqueName: \"kubernetes.io/projected/92466bbf-231a-4554-83e9-136ee0a63a95-kube-api-access-jztgx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.876435 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.876384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztgx\" (UniqueName: \"kubernetes.io/projected/92466bbf-231a-4554-83e9-136ee0a63a95-kube-api-access-jztgx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.876537 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.876445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92466bbf-231a-4554-83e9-136ee0a63a95-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.876820 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.876802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92466bbf-231a-4554-83e9-136ee0a63a95-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.910956 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.910933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztgx\" (UniqueName: \"kubernetes.io/projected/92466bbf-231a-4554-83e9-136ee0a63a95-kube-api-access-jztgx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-56tsq\" (UID: \"92466bbf-231a-4554-83e9-136ee0a63a95\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:25.981026 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:25.980946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b6fb4769-2gf85"]
Sep 18 21:58:25.985200 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:58:25.985177    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf3522_8185_49ea_bbee_4896e5224e64.slice/crio-d67aa41800fb65700338cb6dc64c3b395bdf81f3cd4604972c44a5d4165314ac WatchSource:0}: Error finding container d67aa41800fb65700338cb6dc64c3b395bdf81f3cd4604972c44a5d4165314ac: Status 404 returned error can't find the container with id d67aa41800fb65700338cb6dc64c3b395bdf81f3cd4604972c44a5d4165314ac
Sep 18 21:58:26.186251 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.186213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"
Sep 18 21:58:26.445856 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.445826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b6fb4769-2gf85" event={"ID":"26cf3522-8185-49ea-bbee-4896e5224e64","Type":"ContainerStarted","Data":"fa68d9ddf4baa38f3b8beaff3adf4e24faa10f5acdf4ee0004affe0f5603491f"}
Sep 18 21:58:26.446199 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.445871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b6fb4769-2gf85" event={"ID":"26cf3522-8185-49ea-bbee-4896e5224e64","Type":"ContainerStarted","Data":"d67aa41800fb65700338cb6dc64c3b395bdf81f3cd4604972c44a5d4165314ac"}
Sep 18 21:58:26.518242 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.518167    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67b6fb4769-2gf85" podStartSLOduration=1.518152582 podStartE2EDuration="1.518152582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:26.517240893 +0000 UTC m=+446.715949189" watchObservedRunningTime="2026-09-18 21:58:26.518152582 +0000 UTC m=+446.716860880"
Sep 18 21:58:26.551735 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.551715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq"]
Sep 18 21:58:26.554628 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:58:26.554604    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92466bbf_231a_4554_83e9_136ee0a63a95.slice/crio-d2c456716e2a08bdc99ca9e4ae1a7964a20860f77d1ba13fb0e66e36bd40d2f9 WatchSource:0}: Error finding container d2c456716e2a08bdc99ca9e4ae1a7964a20860f77d1ba13fb0e66e36bd40d2f9: Status 404 returned error can't find the container with id d2c456716e2a08bdc99ca9e4ae1a7964a20860f77d1ba13fb0e66e36bd40d2f9
Sep 18 21:58:26.557131 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:26.557112    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:58:27.451829 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:27.451788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq" event={"ID":"92466bbf-231a-4554-83e9-136ee0a63a95","Type":"ContainerStarted","Data":"d2c456716e2a08bdc99ca9e4ae1a7964a20860f77d1ba13fb0e66e36bd40d2f9"}
Sep 18 21:58:29.460701 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:29.460666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq" event={"ID":"92466bbf-231a-4554-83e9-136ee0a63a95","Type":"ContainerStarted","Data":"5eeefb47cf45cddb33022c16266b423d7f5797a2857aa1140b4a1977349651eb"}
Sep 18 21:58:29.503990 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:29.503911    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-56tsq" podStartSLOduration=1.814457096 podStartE2EDuration="4.503895478s" totalImagesPullingTime="2.689438382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:29.502943133 +0000 UTC m=+449.701651450" watchObservedRunningTime="2026-09-18 21:58:29.503895478 +0000 UTC m=+449.702603775"
Sep 18 21:58:33.190047 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.190011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"]
Sep 18 21:58:33.193044 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.193029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.195825 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.195808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 21:58:33.197139 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.197119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 21:58:33.197206 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.197180    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qmd8c\""
Sep 18 21:58:33.224977 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.224953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"]
Sep 18 21:58:33.227210 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.227182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tlm\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-kube-api-access-q9tlm\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.227286 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.227242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.327713 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.327690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.327806 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.327745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tlm\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-kube-api-access-q9tlm\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.352746 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.352720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tlm\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-kube-api-access-q9tlm\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.356944 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.356915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99759360-4574-42d0-84ce-26fe7c0f10b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5gbdz\" (UID: \"99759360-4574-42d0-84ce-26fe7c0f10b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.502175 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.502123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"
Sep 18 21:58:33.790222 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:33.790159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz"]
Sep 18 21:58:33.796082 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:58:33.796050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99759360_4574_42d0_84ce_26fe7c0f10b8.slice/crio-5bde6bf775347783b54f4abd5abce77d7ae75435c47630a2a72ba5743701d4f5 WatchSource:0}: Error finding container 5bde6bf775347783b54f4abd5abce77d7ae75435c47630a2a72ba5743701d4f5: Status 404 returned error can't find the container with id 5bde6bf775347783b54f4abd5abce77d7ae75435c47630a2a72ba5743701d4f5
Sep 18 21:58:34.479675 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:34.479639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz" event={"ID":"99759360-4574-42d0-84ce-26fe7c0f10b8","Type":"ContainerStarted","Data":"5bde6bf775347783b54f4abd5abce77d7ae75435c47630a2a72ba5743701d4f5"}
Sep 18 21:58:35.696005 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:35.695969    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:35.696486 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:35.696069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:35.701288 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:35.701266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:36.491178 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:36.491150    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67b6fb4769-2gf85"
Sep 18 21:58:36.598056 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:36.598031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:58:37.493066 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:37.493029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz" event={"ID":"99759360-4574-42d0-84ce-26fe7c0f10b8","Type":"ContainerStarted","Data":"72086ae26c7f48680ce5f9989d27c8fbb3bafa2c61333fac353c2c82547ece16"}
Sep 18 21:58:37.533933 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:37.533890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5gbdz" podStartSLOduration=1.8091341459999999 podStartE2EDuration="4.533876928s" totalImagesPullingTime="2.724742782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:37.532844039 +0000 UTC m=+457.731552338" watchObservedRunningTime="2026-09-18 21:58:37.533876928 +0000 UTC m=+457.732585227"
Sep 18 21:58:46.286096 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.286051    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"]
Sep 18 21:58:46.289112 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.289092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.291634 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.291601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rnv7g\""
Sep 18 21:58:46.291747 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.291686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 21:58:46.292228 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.292211    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:58:46.307664 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.307638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"]
Sep 18 21:58:46.326688 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.326668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnql6\" (UniqueName: \"kubernetes.io/projected/f9a1ca22-90f4-4d43-8bd5-f470269823ec-kube-api-access-bnql6\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.326776 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.326713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a1ca22-90f4-4d43-8bd5-f470269823ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.427669 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.427646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnql6\" (UniqueName: \"kubernetes.io/projected/f9a1ca22-90f4-4d43-8bd5-f470269823ec-kube-api-access-bnql6\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.427752 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.427692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a1ca22-90f4-4d43-8bd5-f470269823ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.427959 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.427946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9a1ca22-90f4-4d43-8bd5-f470269823ec-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.452483 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.452460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnql6\" (UniqueName: \"kubernetes.io/projected/f9a1ca22-90f4-4d43-8bd5-f470269823ec-kube-api-access-bnql6\") pod \"openshift-lws-operator-fd8ccff4c-hrsdw\" (UID: \"f9a1ca22-90f4-4d43-8bd5-f470269823ec\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.598774 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.598749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"
Sep 18 21:58:46.875341 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:46.875248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw"]
Sep 18 21:58:46.878298 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:58:46.878277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9a1ca22_90f4_4d43_8bd5_f470269823ec.slice/crio-ead8558d7a86e7568950a34debbb6d5335e1bb2316b22180d625a4229eaaa240 WatchSource:0}: Error finding container ead8558d7a86e7568950a34debbb6d5335e1bb2316b22180d625a4229eaaa240: Status 404 returned error can't find the container with id ead8558d7a86e7568950a34debbb6d5335e1bb2316b22180d625a4229eaaa240
Sep 18 21:58:47.537407 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:47.537354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw" event={"ID":"f9a1ca22-90f4-4d43-8bd5-f470269823ec","Type":"ContainerStarted","Data":"ead8558d7a86e7568950a34debbb6d5335e1bb2316b22180d625a4229eaaa240"}
Sep 18 21:58:49.548513 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:49.548430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw" event={"ID":"f9a1ca22-90f4-4d43-8bd5-f470269823ec","Type":"ContainerStarted","Data":"13974c22b2678d082ece5a166a5effe3cf83a5de4ed98ec847de67d119c04561"}
Sep 18 21:58:49.574769 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:58:49.574713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hrsdw" podStartSLOduration=1.186774579 podStartE2EDuration="3.57469572s" totalImagesPullingTime="2.387921141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:58:49.573705675 +0000 UTC m=+469.772413977" watchObservedRunningTime="2026-09-18 21:58:49.57469572 +0000 UTC m=+469.773404022"
Sep 18 21:59:01.616409 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.616339    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-676985698c-5pvlz" podUID="3a605bff-bac1-47cd-87d8-fcb2668e174d" containerName="console" containerID="cri-o://bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926" gracePeriod=15
Sep 18 21:59:01.868054 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.868002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:59:01.935044 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935152 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935152 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935152 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935287 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htmkg\" (UniqueName: \"kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935339 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935430 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935385    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca\") pod \"3a605bff-bac1-47cd-87d8-fcb2668e174d\" (UID: \"3a605bff-bac1-47cd-87d8-fcb2668e174d\") "
Sep 18 21:59:01.935504 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:59:01.935557 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config" (OuterVolumeSpecName: "console-config") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:59:01.935603 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:59:01.935760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:59:01.935760 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935753    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-trusted-ca-bundle\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:01.935873 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935773    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:01.935873 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.935787    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-oauth-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:01.937214 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.937192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg" (OuterVolumeSpecName: "kube-api-access-htmkg") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "kube-api-access-htmkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:59:01.937214 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.937201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:59:01.937363 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:01.937273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a605bff-bac1-47cd-87d8-fcb2668e174d" (UID: "3a605bff-bac1-47cd-87d8-fcb2668e174d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:59:02.036767 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.036740    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmkg\" (UniqueName: \"kubernetes.io/projected/3a605bff-bac1-47cd-87d8-fcb2668e174d-kube-api-access-htmkg\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:02.036887 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.036771    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-serving-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:02.036887 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.036785    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a605bff-bac1-47cd-87d8-fcb2668e174d-service-ca\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:02.036887 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.036797    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a605bff-bac1-47cd-87d8-fcb2668e174d-console-oauth-config\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 21:59:02.277434 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.277347    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 21:59:02.277618 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.277606    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a605bff-bac1-47cd-87d8-fcb2668e174d" containerName="console"
Sep 18 21:59:02.277665 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.277620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a605bff-bac1-47cd-87d8-fcb2668e174d" containerName="console"
Sep 18 21:59:02.277699 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.277672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a605bff-bac1-47cd-87d8-fcb2668e174d" containerName="console"
Sep 18 21:59:02.317262 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.317235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 21:59:02.317383 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.317328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.319812 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.319790    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 21:59:02.320336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.320320    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 21:59:02.320828 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.320810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 21:59:02.322137 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.322121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 21:59:02.322331 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.322311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 21:59:02.322449 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.322315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 21:59:02.322449 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.322354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rnx9w\""
Sep 18 21:59:02.440220 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440324 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440324 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440324 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440477 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440477 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.440477 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.440435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggw4\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541122 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541122 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541122 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541122 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541429 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggw4\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541429 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541429 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.541891 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.541844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.543327 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.543306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.543494 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.543475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.543592 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.543571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.543630 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.543600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.568768 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.568742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.570009 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.569988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggw4\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4\") pod \"istiod-openshift-gateway-7b7995d5cd-9jj29\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.600013 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.599986    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a605bff-bac1-47cd-87d8-fcb2668e174d" containerID="bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926" exitCode=0
Sep 18 21:59:02.600098 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.600025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676985698c-5pvlz"
Sep 18 21:59:02.600098 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.600032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676985698c-5pvlz" event={"ID":"3a605bff-bac1-47cd-87d8-fcb2668e174d","Type":"ContainerDied","Data":"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"}
Sep 18 21:59:02.600098 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.600077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676985698c-5pvlz" event={"ID":"3a605bff-bac1-47cd-87d8-fcb2668e174d","Type":"ContainerDied","Data":"70b31eb7c8ed48b29ccbad9cfb5610cc49e98b32539720a81f933b2b113b3536"}
Sep 18 21:59:02.600098 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.600098    2563 scope.go:117] "RemoveContainer" containerID="bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"
Sep 18 21:59:02.608001 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.607972    2563 scope.go:117] "RemoveContainer" containerID="bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"
Sep 18 21:59:02.608284 ip-10-0-141-13 kubenswrapper[2563]: E0918 21:59:02.608264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926\": container with ID starting with bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926 not found: ID does not exist" containerID="bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"
Sep 18 21:59:02.608336 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.608292    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926"} err="failed to get container status \"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926\": rpc error: code = NotFound desc = could not find container \"bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926\": container with ID starting with bf0d93894445f5d92dc4610c85f5dd7f2e240566856382f47603d195a3440926 not found: ID does not exist"
Sep 18 21:59:02.626569 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.626552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:02.677621 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.677602    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:59:02.690940 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.690919    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-676985698c-5pvlz"]
Sep 18 21:59:02.943887 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:02.943864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 21:59:02.945673 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:59:02.945646    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507d1f38_9b01_49f8_b99b_0f437a47ed99.slice/crio-dae18f1474da3c5a3975cf9a7d80c101e3e279c02bf2e0d75e9b106e65bc4bb6 WatchSource:0}: Error finding container dae18f1474da3c5a3975cf9a7d80c101e3e279c02bf2e0d75e9b106e65bc4bb6: Status 404 returned error can't find the container with id dae18f1474da3c5a3975cf9a7d80c101e3e279c02bf2e0d75e9b106e65bc4bb6
Sep 18 21:59:03.605376 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:03.605343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" event={"ID":"507d1f38-9b01-49f8-b99b-0f437a47ed99","Type":"ContainerStarted","Data":"dae18f1474da3c5a3975cf9a7d80c101e3e279c02bf2e0d75e9b106e65bc4bb6"}
Sep 18 21:59:03.883289 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:03.883211    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a605bff-bac1-47cd-87d8-fcb2668e174d" path="/var/lib/kubelet/pods/3a605bff-bac1-47cd-87d8-fcb2668e174d/volumes"
Sep 18 21:59:05.514070 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:05.514022    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:59:05.514374 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:05.514094    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:59:05.617625 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:05.617594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" event={"ID":"507d1f38-9b01-49f8-b99b-0f437a47ed99","Type":"ContainerStarted","Data":"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"}
Sep 18 21:59:05.617781 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:05.617721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:06.626264 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:06.626222    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9jj29 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 21:59:06.626806 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:06.626282    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 21:59:09.625403 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:09.625377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 21:59:09.677612 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:09.674984    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" podStartSLOduration=5.109034406 podStartE2EDuration="7.674952808s" totalImagesPullingTime="2.565918402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:05.646980422 +0000 UTC m=+485.845688722" watchObservedRunningTime="2026-09-18 21:59:09.674952808 +0000 UTC m=+489.873661108"
Sep 18 21:59:36.528000 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.527968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"]
Sep 18 21:59:36.536121 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.536101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:36.539180 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.539154    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 21:59:36.539383 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.539361    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 21:59:36.539458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.539361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vkw6t\""
Sep 18 21:59:36.539458 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.539447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 21:59:36.550839 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.550811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"]
Sep 18 21:59:36.689329 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.689300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lvb\" (UniqueName: \"kubernetes.io/projected/7c2d9e8c-5c3d-49ad-91fb-de024c657704-kube-api-access-j9lvb\") pod \"dns-operator-controller-manager-65dc8f76f5-4sgbx\" (UID: \"7c2d9e8c-5c3d-49ad-91fb-de024c657704\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:36.789840 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.789786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lvb\" (UniqueName: \"kubernetes.io/projected/7c2d9e8c-5c3d-49ad-91fb-de024c657704-kube-api-access-j9lvb\") pod \"dns-operator-controller-manager-65dc8f76f5-4sgbx\" (UID: \"7c2d9e8c-5c3d-49ad-91fb-de024c657704\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:36.803541 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.803522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lvb\" (UniqueName: \"kubernetes.io/projected/7c2d9e8c-5c3d-49ad-91fb-de024c657704-kube-api-access-j9lvb\") pod \"dns-operator-controller-manager-65dc8f76f5-4sgbx\" (UID: \"7c2d9e8c-5c3d-49ad-91fb-de024c657704\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:36.846443 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:36.846424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:37.130864 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:37.130842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"]
Sep 18 21:59:37.133342 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:59:37.133315    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2d9e8c_5c3d_49ad_91fb_de024c657704.slice/crio-2c4410411ec7f2abd39c037643abd3a9a33883726a34c545e9cdb6e785fdda04 WatchSource:0}: Error finding container 2c4410411ec7f2abd39c037643abd3a9a33883726a34c545e9cdb6e785fdda04: Status 404 returned error can't find the container with id 2c4410411ec7f2abd39c037643abd3a9a33883726a34c545e9cdb6e785fdda04
Sep 18 21:59:37.751085 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:37.751051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx" event={"ID":"7c2d9e8c-5c3d-49ad-91fb-de024c657704","Type":"ContainerStarted","Data":"2c4410411ec7f2abd39c037643abd3a9a33883726a34c545e9cdb6e785fdda04"}
Sep 18 21:59:39.689377 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.689352    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-4srxn"]
Sep 18 21:59:39.692339 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.692322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:39.696103 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.696083    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w7jkd\""
Sep 18 21:59:39.726327 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.726297    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-4srxn"]
Sep 18 21:59:39.760489 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.760463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx" event={"ID":"7c2d9e8c-5c3d-49ad-91fb-de024c657704","Type":"ContainerStarted","Data":"3332db74652a6532c0a808079120b9ff4baa876e233262eb1843ff3d077884be"}
Sep 18 21:59:39.760605 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.760589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:39.817414 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.817340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrtf\" (UniqueName: \"kubernetes.io/projected/4384570e-0e13-4d35-9708-32fc77e1bbb1-kube-api-access-ctrtf\") pod \"authorino-operator-6fc8f864f4-4srxn\" (UID: \"4384570e-0e13-4d35-9708-32fc77e1bbb1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:39.917914 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.917881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrtf\" (UniqueName: \"kubernetes.io/projected/4384570e-0e13-4d35-9708-32fc77e1bbb1-kube-api-access-ctrtf\") pod \"authorino-operator-6fc8f864f4-4srxn\" (UID: \"4384570e-0e13-4d35-9708-32fc77e1bbb1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:39.930157 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:39.930129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrtf\" (UniqueName: \"kubernetes.io/projected/4384570e-0e13-4d35-9708-32fc77e1bbb1-kube-api-access-ctrtf\") pod \"authorino-operator-6fc8f864f4-4srxn\" (UID: \"4384570e-0e13-4d35-9708-32fc77e1bbb1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:40.001785 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:40.001767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:40.306753 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:40.306701    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx" podStartSLOduration=1.90376809 podStartE2EDuration="4.306680429s" totalImagesPullingTime="2.402912339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:39.789477717 +0000 UTC m=+519.988186016" watchObservedRunningTime="2026-09-18 21:59:40.306680429 +0000 UTC m=+520.505388727"
Sep 18 21:59:40.306958 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:40.306940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-4srxn"]
Sep 18 21:59:40.315029 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:59:40.315001    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4384570e_0e13_4d35_9708_32fc77e1bbb1.slice/crio-d6ddee4dd29044029dd85047549a14b7410bffdbc43ed618b21a2a0d15b7df94 WatchSource:0}: Error finding container d6ddee4dd29044029dd85047549a14b7410bffdbc43ed618b21a2a0d15b7df94: Status 404 returned error can't find the container with id d6ddee4dd29044029dd85047549a14b7410bffdbc43ed618b21a2a0d15b7df94
Sep 18 21:59:40.766131 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:40.766099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn" event={"ID":"4384570e-0e13-4d35-9708-32fc77e1bbb1","Type":"ContainerStarted","Data":"d6ddee4dd29044029dd85047549a14b7410bffdbc43ed618b21a2a0d15b7df94"}
Sep 18 21:59:42.778730 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:42.778687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn" event={"ID":"4384570e-0e13-4d35-9708-32fc77e1bbb1","Type":"ContainerStarted","Data":"a6e1fb0d2174a0ca16d01f2f97614d09c9ee97d8046916fa0609212b42392d47"}
Sep 18 21:59:42.779114 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:42.778761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:42.805857 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:42.805810    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn" podStartSLOduration=2.40228413 podStartE2EDuration="3.80579504s" totalImagesPullingTime="1.40351091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:42.805203319 +0000 UTC m=+523.003911618" watchObservedRunningTime="2026-09-18 21:59:42.80579504 +0000 UTC m=+523.004503339"
Sep 18 21:59:45.233387 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.233311    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"]
Sep 18 21:59:45.236641 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.236622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.242764 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.242746    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8dmts\""
Sep 18 21:59:45.266198 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.266173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"]
Sep 18 21:59:45.355486 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.355458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-kube-api-access-v8zvx\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.355585 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.355505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.456294 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.456269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-kube-api-access-v8zvx\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.456388 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.456314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.456713 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.456695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.471209 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.471183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zvx\" (UniqueName: \"kubernetes.io/projected/d161bb61-04e8-4fa7-bbeb-15ea3bec964f-kube-api-access-v8zvx\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vfzqt\" (UID: \"d161bb61-04e8-4fa7-bbeb-15ea3bec964f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.545822 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.545734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:45.895117 ip-10-0-141-13 kubenswrapper[2563]: W0918 21:59:45.895085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd161bb61_04e8_4fa7_bbeb_15ea3bec964f.slice/crio-ce4bf3d10aee47a17d811ee77db4d24f1d6ac6be332a10798b7b49e2428f0857 WatchSource:0}: Error finding container ce4bf3d10aee47a17d811ee77db4d24f1d6ac6be332a10798b7b49e2428f0857: Status 404 returned error can't find the container with id ce4bf3d10aee47a17d811ee77db4d24f1d6ac6be332a10798b7b49e2428f0857
Sep 18 21:59:45.896095 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:45.896073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"]
Sep 18 21:59:46.795820 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:46.795786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt" event={"ID":"d161bb61-04e8-4fa7-bbeb-15ea3bec964f","Type":"ContainerStarted","Data":"ce4bf3d10aee47a17d811ee77db4d24f1d6ac6be332a10798b7b49e2428f0857"}
Sep 18 21:59:50.768807 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:50.768771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4sgbx"
Sep 18 21:59:53.785295 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:53.785265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-4srxn"
Sep 18 21:59:59.852025 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:59.851992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt" event={"ID":"d161bb61-04e8-4fa7-bbeb-15ea3bec964f","Type":"ContainerStarted","Data":"b59e285ec7a885b6822f0661638c893a08d2b6c1bd9b7ad3bf66da27696cd1e9"}
Sep 18 21:59:59.852388 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:59.852066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 21:59:59.884673 ip-10-0-141-13 kubenswrapper[2563]: I0918 21:59:59.884621    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt" podStartSLOduration=1.739437153 podStartE2EDuration="14.884603813s" totalImagesPullingTime="13.14516666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:59.882759627 +0000 UTC m=+540.081467925" watchObservedRunningTime="2026-09-18 21:59:59.884603813 +0000 UTC m=+540.083312114"
Sep 18 22:00:10.858105 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:10.858069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vfzqt"
Sep 18 22:00:44.541731 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.541699    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:44.543789 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.543773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.547758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.547735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wmx9d\""
Sep 18 22:00:44.547858 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.547735    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 22:00:44.558935 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.558914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:44.628437 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.628384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:44.671435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.671390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckr6\" (UniqueName: \"kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.671435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.671443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.772847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.772818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckr6\" (UniqueName: \"kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.772847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.772850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.773437 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.773419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.781602 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.781585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckr6\" (UniqueName: \"kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6\") pod \"limitador-limitador-8588749977-whjnb\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") " pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:44.853100 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:44.853076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:45.125788 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:45.125659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:45.128235 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:00:45.128205    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447cbdae_185d_40a5_a9a7_6fda62fc7e14.slice/crio-9d7287b6175c227bed81cc470b74beaeead9e52bbbdac732d756ef5168951bc0 WatchSource:0}: Error finding container 9d7287b6175c227bed81cc470b74beaeead9e52bbbdac732d756ef5168951bc0: Status 404 returned error can't find the container with id 9d7287b6175c227bed81cc470b74beaeead9e52bbbdac732d756ef5168951bc0
Sep 18 22:00:46.031162 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:46.031124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" event={"ID":"447cbdae-185d-40a5-a9a7-6fda62fc7e14","Type":"ContainerStarted","Data":"9d7287b6175c227bed81cc470b74beaeead9e52bbbdac732d756ef5168951bc0"}
Sep 18 22:00:47.036335 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:47.036302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" event={"ID":"447cbdae-185d-40a5-a9a7-6fda62fc7e14","Type":"ContainerStarted","Data":"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"}
Sep 18 22:00:47.036723 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:47.036426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:47.079372 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:47.079329    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" podStartSLOduration=1.9536493689999999 podStartE2EDuration="3.079315194s" totalImagesPullingTime="1.125665825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:47.078480427 +0000 UTC m=+587.277188727" watchObservedRunningTime="2026-09-18 22:00:47.079315194 +0000 UTC m=+587.278023559"
Sep 18 22:00:58.041346 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.041319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:58.207621 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.207588    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:58.207797 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.207759    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" podUID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" containerName="limitador" containerID="cri-o://48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709" gracePeriod=30
Sep 18 22:00:58.741179 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.741157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:58.878033 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.878010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file\") pod \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") "
Sep 18 22:00:58.878154 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.878052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pckr6\" (UniqueName: \"kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6\") pod \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\" (UID: \"447cbdae-185d-40a5-a9a7-6fda62fc7e14\") "
Sep 18 22:00:58.878343 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.878322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file" (OuterVolumeSpecName: "config-file") pod "447cbdae-185d-40a5-a9a7-6fda62fc7e14" (UID: "447cbdae-185d-40a5-a9a7-6fda62fc7e14"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:00:58.880188 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.880170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6" (OuterVolumeSpecName: "kube-api-access-pckr6") pod "447cbdae-185d-40a5-a9a7-6fda62fc7e14" (UID: "447cbdae-185d-40a5-a9a7-6fda62fc7e14"). InnerVolumeSpecName "kube-api-access-pckr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:00:58.978865 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.978844    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/447cbdae-185d-40a5-a9a7-6fda62fc7e14-config-file\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:00:58.978865 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:58.978862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckr6\" (UniqueName: \"kubernetes.io/projected/447cbdae-185d-40a5-a9a7-6fda62fc7e14-kube-api-access-pckr6\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:00:59.086646 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.086619    2563 generic.go:356] "Generic (PLEG): container finished" podID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" containerID="48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709" exitCode=0
Sep 18 22:00:59.086937 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.086663    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-whjnb"
Sep 18 22:00:59.086937 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.086663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" event={"ID":"447cbdae-185d-40a5-a9a7-6fda62fc7e14","Type":"ContainerDied","Data":"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"}
Sep 18 22:00:59.086937 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.086714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-whjnb" event={"ID":"447cbdae-185d-40a5-a9a7-6fda62fc7e14","Type":"ContainerDied","Data":"9d7287b6175c227bed81cc470b74beaeead9e52bbbdac732d756ef5168951bc0"}
Sep 18 22:00:59.086937 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.086734    2563 scope.go:117] "RemoveContainer" containerID="48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"
Sep 18 22:00:59.095002 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.094986    2563 scope.go:117] "RemoveContainer" containerID="48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"
Sep 18 22:00:59.095257 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:00:59.095241    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709\": container with ID starting with 48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709 not found: ID does not exist" containerID="48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"
Sep 18 22:00:59.095307 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.095264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709"} err="failed to get container status \"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709\": rpc error: code = NotFound desc = could not find container \"48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709\": container with ID starting with 48d7930c0b9f3a3455343a1fc41f3f1ba2e14265b8e6027e89da5e6ee91aa709 not found: ID does not exist"
Sep 18 22:00:59.110352 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.110331    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:59.114707 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.114683    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-whjnb"]
Sep 18 22:00:59.881974 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:00:59.881948    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" path="/var/lib/kubelet/pods/447cbdae-185d-40a5-a9a7-6fda62fc7e14/volumes"
Sep 18 22:01:01.834693 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:01.834664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:01:01.835127 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:01.834929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:01:08.049286 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.049252    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7df2d"]
Sep 18 22:01:08.049754 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.049657    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" containerName="limitador"
Sep 18 22:01:08.049754 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.049675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" containerName="limitador"
Sep 18 22:01:08.049754 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.049742    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="447cbdae-185d-40a5-a9a7-6fda62fc7e14" containerName="limitador"
Sep 18 22:01:08.053318 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.053298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.056417 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.056387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 22:01:08.056528 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.056419    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkzfw\""
Sep 18 22:01:08.101995 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.101969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7df2d"]
Sep 18 22:01:08.138791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.138766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70b36827-60f5-46e3-aefb-c52dd57fbcbb-tls-cert\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.138898 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.138803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px74s\" (UniqueName: \"kubernetes.io/projected/70b36827-60f5-46e3-aefb-c52dd57fbcbb-kube-api-access-px74s\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.239159 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.239136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70b36827-60f5-46e3-aefb-c52dd57fbcbb-tls-cert\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.239258 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.239172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px74s\" (UniqueName: \"kubernetes.io/projected/70b36827-60f5-46e3-aefb-c52dd57fbcbb-kube-api-access-px74s\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.241687 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.241666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/70b36827-60f5-46e3-aefb-c52dd57fbcbb-tls-cert\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.250166 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.250146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px74s\" (UniqueName: \"kubernetes.io/projected/70b36827-60f5-46e3-aefb-c52dd57fbcbb-kube-api-access-px74s\") pod \"authorino-6d9898bd99-7df2d\" (UID: \"70b36827-60f5-46e3-aefb-c52dd57fbcbb\") " pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.362266 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.362245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-7df2d"
Sep 18 22:01:08.636692 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:08.636664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-7df2d"]
Sep 18 22:01:08.638318 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:01:08.638282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b36827_60f5_46e3_aefb_c52dd57fbcbb.slice/crio-8237199eca9d7c2b40b5ecce8a056316d6436fc2fcdeddf04dfa6a471458de73 WatchSource:0}: Error finding container 8237199eca9d7c2b40b5ecce8a056316d6436fc2fcdeddf04dfa6a471458de73: Status 404 returned error can't find the container with id 8237199eca9d7c2b40b5ecce8a056316d6436fc2fcdeddf04dfa6a471458de73
Sep 18 22:01:09.129295 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:09.129265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-7df2d" event={"ID":"70b36827-60f5-46e3-aefb-c52dd57fbcbb","Type":"ContainerStarted","Data":"8237199eca9d7c2b40b5ecce8a056316d6436fc2fcdeddf04dfa6a471458de73"}
Sep 18 22:01:10.134736 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:10.134708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-7df2d" event={"ID":"70b36827-60f5-46e3-aefb-c52dd57fbcbb","Type":"ContainerStarted","Data":"064e19054edfa62c1ded4d230df57450c5ee71ccbdb8d8a3bdd5f9d391405dd6"}
Sep 18 22:01:10.162010 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:10.161963    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-7df2d" podStartSLOduration=0.746533759 podStartE2EDuration="2.161946585s" totalImagesPullingTime="1.415412826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:10.160455512 +0000 UTC m=+610.359163810" watchObservedRunningTime="2026-09-18 22:01:10.161946585 +0000 UTC m=+610.360654883"
Sep 18 22:01:31.028218 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.028184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"]
Sep 18 22:01:31.032222 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.032204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.055118 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.055093    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"]
Sep 18 22:01:31.194389 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194520 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194520 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194520 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194520 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae4f53-da2d-4e99-ac77-cb536497b251-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194520 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b97g\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-kube-api-access-8b97g\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.194682 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.194528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295537 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295537 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295537 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae4f53-da2d-4e99-ac77-cb536497b251-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.295744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.295621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b97g\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-kube-api-access-8b97g\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.296304 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.296278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.297880 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.297853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae4f53-da2d-4e99-ac77-cb536497b251-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.297961 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.297894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.298036 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.298017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.298072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.298031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.307607 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.307581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b97g\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-kube-api-access-8b97g\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.307895 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.307878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae4f53-da2d-4e99-ac77-cb536497b251-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zhst7\" (UID: \"a1ae4f53-da2d-4e99-ac77-cb536497b251\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.341383 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.341361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:31.626344 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.626315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"]
Sep 18 22:01:31.630673 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:01:31.630647    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ae4f53_da2d_4e99_ac77_cb536497b251.slice/crio-a12ec1d915ccec66cb61abb2d10d877ab7f28fcd4b926fd9fa6f57defdf4f6a2 WatchSource:0}: Error finding container a12ec1d915ccec66cb61abb2d10d877ab7f28fcd4b926fd9fa6f57defdf4f6a2: Status 404 returned error can't find the container with id a12ec1d915ccec66cb61abb2d10d877ab7f28fcd4b926fd9fa6f57defdf4f6a2
Sep 18 22:01:31.632624 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.632592    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:01:31.632708 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:31.632659    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:01:32.226535 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.226500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7" event={"ID":"a1ae4f53-da2d-4e99-ac77-cb536497b251","Type":"ContainerStarted","Data":"7b075b3d47a0b29a21171aa060502b558dc3e1bb365d70bec15249c7cfb0a27e"}
Sep 18 22:01:32.226890 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.226587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7" event={"ID":"a1ae4f53-da2d-4e99-ac77-cb536497b251","Type":"ContainerStarted","Data":"a12ec1d915ccec66cb61abb2d10d877ab7f28fcd4b926fd9fa6f57defdf4f6a2"}
Sep 18 22:01:32.226890 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.226723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:32.228341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.228316    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-zhst7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:01:32.228483 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.228378    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7" podUID="a1ae4f53-da2d-4e99-ac77-cb536497b251" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:01:32.250847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:32.250799    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7" podStartSLOduration=1.250782236 podStartE2EDuration="1.250782236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:32.248152699 +0000 UTC m=+632.446860999" watchObservedRunningTime="2026-09-18 22:01:32.250782236 +0000 UTC m=+632.449490537"
Sep 18 22:01:33.232265 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.232237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zhst7"
Sep 18 22:01:33.317449 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.317421    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 22:01:33.317628 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.317609    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerName="discovery" containerID="cri-o://9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288" gracePeriod=30
Sep 18 22:01:33.554219 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.554198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 22:01:33.714428 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714428 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ggw4\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.714651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714562    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig\") pod \"507d1f38-9b01-49f8-b99b-0f437a47ed99\" (UID: \"507d1f38-9b01-49f8-b99b-0f437a47ed99\") "
Sep 18 22:01:33.715035 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.714990    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:01:33.716898 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.716869    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs" (OuterVolumeSpecName: "local-certs") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:01:33.717024 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.716999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4" (OuterVolumeSpecName: "kube-api-access-6ggw4") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "kube-api-access-6ggw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:01:33.717106 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.717039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts" (OuterVolumeSpecName: "cacerts") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:01:33.717106 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.717096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:01:33.717207 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.717185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token" (OuterVolumeSpecName: "istio-token") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:01:33.717243 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.717207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "507d1f38-9b01-49f8-b99b-0f437a47ed99" (UID: "507d1f38-9b01-49f8-b99b-0f437a47ed99"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:01:33.815527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815478    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-kubeconfig\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815497    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-dns-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815506    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggw4\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-kube-api-access-6ggw4\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815517    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-csr-ca-configmap\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815525    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/507d1f38-9b01-49f8-b99b-0f437a47ed99-cacerts\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815717 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815533    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/507d1f38-9b01-49f8-b99b-0f437a47ed99-local-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:33.815717 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:33.815542    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507d1f38-9b01-49f8-b99b-0f437a47ed99-istio-token\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:01:34.237319 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.237283    2563 generic.go:356] "Generic (PLEG): container finished" podID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerID="9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288" exitCode=0
Sep 18 22:01:34.237768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.237332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"
Sep 18 22:01:34.237768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.237377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" event={"ID":"507d1f38-9b01-49f8-b99b-0f437a47ed99","Type":"ContainerDied","Data":"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"}
Sep 18 22:01:34.237768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.237444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29" event={"ID":"507d1f38-9b01-49f8-b99b-0f437a47ed99","Type":"ContainerDied","Data":"dae18f1474da3c5a3975cf9a7d80c101e3e279c02bf2e0d75e9b106e65bc4bb6"}
Sep 18 22:01:34.237768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.237465    2563 scope.go:117] "RemoveContainer" containerID="9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"
Sep 18 22:01:34.245648 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.245631    2563 scope.go:117] "RemoveContainer" containerID="9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"
Sep 18 22:01:34.245906 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:01:34.245886    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288\": container with ID starting with 9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288 not found: ID does not exist" containerID="9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"
Sep 18 22:01:34.245955 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.245916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288"} err="failed to get container status \"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288\": rpc error: code = NotFound desc = could not find container \"9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288\": container with ID starting with 9c694cb4ad13ee477f7907672a32547c8d5e62c20514557485e09cb065ef2288 not found: ID does not exist"
Sep 18 22:01:34.274492 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.274470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 22:01:34.280365 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:34.280346    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9jj29"]
Sep 18 22:01:35.881316 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:35.881284    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" path="/var/lib/kubelet/pods/507d1f38-9b01-49f8-b99b-0f437a47ed99/volumes"
Sep 18 22:01:45.523707 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.523676    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:01:45.524050 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.523938    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerName="discovery"
Sep 18 22:01:45.524050 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.523949    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerName="discovery"
Sep 18 22:01:45.524050 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.523993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="507d1f38-9b01-49f8-b99b-0f437a47ed99" containerName="discovery"
Sep 18 22:01:45.531141 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.531118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.533874 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.533855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 18 22:01:45.534382 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.534363    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 22:01:45.534748 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.534733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-sk9zs\""
Sep 18 22:01:45.534972 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.534745    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 22:01:45.539838 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.539817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:01:45.563894 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.563868    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7774f8b66-9w5pn"]
Sep 18 22:01:45.566823 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.566808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.570183 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.570165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 22:01:45.570379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.570360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-jv5d6\""
Sep 18 22:01:45.581328 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.581308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7774f8b66-9w5pn"]
Sep 18 22:01:45.594308 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.594284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.594435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.594311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7t4h\" (UniqueName: \"kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.594435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.594410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crfqx\" (UniqueName: \"kubernetes.io/projected/c4766513-3605-433b-b3f1-2c7b0e0cd98a-kube-api-access-crfqx\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.594558 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.594454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4766513-3605-433b-b3f1-2c7b0e0cd98a-cert\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.694940 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.694908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crfqx\" (UniqueName: \"kubernetes.io/projected/c4766513-3605-433b-b3f1-2c7b0e0cd98a-kube-api-access-crfqx\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.695054 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.694944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4766513-3605-433b-b3f1-2c7b0e0cd98a-cert\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.695113 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.695068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.695113 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.695100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7t4h\" (UniqueName: \"kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.695201 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:01:45.695182    2563 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 22:01:45.695268 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:01:45.695256    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert podName:55f55e16-783f-41b4-802f-a73027f60c2d nodeName:}" failed. No retries permitted until 2026-09-18 22:01:46.195240036 +0000 UTC m=+646.393948313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert") pod "kserve-controller-manager-575575dfd7-mvh9l" (UID: "55f55e16-783f-41b4-802f-a73027f60c2d") : secret "kserve-webhook-server-cert" not found
Sep 18 22:01:45.697420 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.697385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4766513-3605-433b-b3f1-2c7b0e0cd98a-cert\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.719201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.719180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7t4h\" (UniqueName: \"kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:45.719406 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.719376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crfqx\" (UniqueName: \"kubernetes.io/projected/c4766513-3605-433b-b3f1-2c7b0e0cd98a-kube-api-access-crfqx\") pod \"llmisvc-controller-manager-7774f8b66-9w5pn\" (UID: \"c4766513-3605-433b-b3f1-2c7b0e0cd98a\") " pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:45.876472 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:45.876451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:46.152437 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.152343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7774f8b66-9w5pn"]
Sep 18 22:01:46.155151 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:01:46.155119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4766513_3605_433b_b3f1_2c7b0e0cd98a.slice/crio-51288bc7d1da5e14b1a1c7f0e9ba2baed7e34e5b03f721123b648302e2aa6abc WatchSource:0}: Error finding container 51288bc7d1da5e14b1a1c7f0e9ba2baed7e34e5b03f721123b648302e2aa6abc: Status 404 returned error can't find the container with id 51288bc7d1da5e14b1a1c7f0e9ba2baed7e34e5b03f721123b648302e2aa6abc
Sep 18 22:01:46.199436 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.199412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:46.201671 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.201651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") pod \"kserve-controller-manager-575575dfd7-mvh9l\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") " pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:46.287292 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.287261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn" event={"ID":"c4766513-3605-433b-b3f1-2c7b0e0cd98a","Type":"ContainerStarted","Data":"51288bc7d1da5e14b1a1c7f0e9ba2baed7e34e5b03f721123b648302e2aa6abc"}
Sep 18 22:01:46.441983 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.441927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:46.715708 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:46.715640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:01:46.717789 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:01:46.717764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f55e16_783f_41b4_802f_a73027f60c2d.slice/crio-adfde758df04155062d85606d196211c136c023a67779bd2e356aee5a190610f WatchSource:0}: Error finding container adfde758df04155062d85606d196211c136c023a67779bd2e356aee5a190610f: Status 404 returned error can't find the container with id adfde758df04155062d85606d196211c136c023a67779bd2e356aee5a190610f
Sep 18 22:01:47.293988 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:47.293934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" event={"ID":"55f55e16-783f-41b4-802f-a73027f60c2d","Type":"ContainerStarted","Data":"adfde758df04155062d85606d196211c136c023a67779bd2e356aee5a190610f"}
Sep 18 22:01:50.310364 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.310323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" event={"ID":"55f55e16-783f-41b4-802f-a73027f60c2d","Type":"ContainerStarted","Data":"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"}
Sep 18 22:01:50.310799 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.310447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:01:50.311951 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.311929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn" event={"ID":"c4766513-3605-433b-b3f1-2c7b0e0cd98a","Type":"ContainerStarted","Data":"e9d6449ba7ac8fde641c93cdc1812f6eb4706bd97386dfaf8197874f559b884b"}
Sep 18 22:01:50.312043 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.311981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:01:50.335568 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.335527    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" podStartSLOduration=2.517212046 podStartE2EDuration="5.335514184s" totalImagesPullingTime="2.818302138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:50.333933397 +0000 UTC m=+650.532641697" watchObservedRunningTime="2026-09-18 22:01:50.335514184 +0000 UTC m=+650.534222518"
Sep 18 22:01:50.357550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:01:50.357510    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn" podStartSLOduration=1.979628689 podStartE2EDuration="5.357496537s" totalImagesPullingTime="3.377867848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:50.355688138 +0000 UTC m=+650.554396437" watchObservedRunningTime="2026-09-18 22:01:50.357496537 +0000 UTC m=+650.556204836"
Sep 18 22:02:21.318349 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:21.318318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7774f8b66-9w5pn"
Sep 18 22:02:21.321387 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:21.321366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:02:31.403298 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.403261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:02:31.403781 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.403493    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" podUID="55f55e16-783f-41b4-802f-a73027f60c2d" containerName="manager" containerID="cri-o://ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4" gracePeriod=10
Sep 18 22:02:31.436065 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.436044    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-nrvzj"]
Sep 18 22:02:31.438272 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.438258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.457775 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.457745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-nrvzj"]
Sep 18 22:02:31.519836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.519813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdz6\" (UniqueName: \"kubernetes.io/projected/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-kube-api-access-hrdz6\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.519917 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.519849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-cert\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.620363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.620333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdz6\" (UniqueName: \"kubernetes.io/projected/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-kube-api-access-hrdz6\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.620521 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.620373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-cert\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.622568 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.622545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-cert\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.637893 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.637876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdz6\" (UniqueName: \"kubernetes.io/projected/6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6-kube-api-access-hrdz6\") pod \"kserve-controller-manager-575575dfd7-nrvzj\" (UID: \"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6\") " pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.648784 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.648766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:02:31.720727 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.720675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7t4h\" (UniqueName: \"kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h\") pod \"55f55e16-783f-41b4-802f-a73027f60c2d\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") "
Sep 18 22:02:31.720727 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.720714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") pod \"55f55e16-783f-41b4-802f-a73027f60c2d\" (UID: \"55f55e16-783f-41b4-802f-a73027f60c2d\") "
Sep 18 22:02:31.722769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.722716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h" (OuterVolumeSpecName: "kube-api-access-n7t4h") pod "55f55e16-783f-41b4-802f-a73027f60c2d" (UID: "55f55e16-783f-41b4-802f-a73027f60c2d"). InnerVolumeSpecName "kube-api-access-n7t4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:02:31.722870 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.722823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert" (OuterVolumeSpecName: "cert") pod "55f55e16-783f-41b4-802f-a73027f60c2d" (UID: "55f55e16-783f-41b4-802f-a73027f60c2d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:02:31.789158 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.789137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:31.821590 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.821560    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7t4h\" (UniqueName: \"kubernetes.io/projected/55f55e16-783f-41b4-802f-a73027f60c2d-kube-api-access-n7t4h\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:02:31.821590 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:31.821585    2563 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55f55e16-783f-41b4-802f-a73027f60c2d-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:02:32.066229 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.066204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-nrvzj"]
Sep 18 22:02:32.068113 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:02:32.068084    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0d1682_4a41_4f3f_a6dc_2fe26d17aff6.slice/crio-812fe3df4fbb6b5a033e927dd70d9c133a77ec2a5d218645a1835a5f5c606fc8 WatchSource:0}: Error finding container 812fe3df4fbb6b5a033e927dd70d9c133a77ec2a5d218645a1835a5f5c606fc8: Status 404 returned error can't find the container with id 812fe3df4fbb6b5a033e927dd70d9c133a77ec2a5d218645a1835a5f5c606fc8
Sep 18 22:02:32.490660 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.490631    2563 generic.go:356] "Generic (PLEG): container finished" podID="55f55e16-783f-41b4-802f-a73027f60c2d" containerID="ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4" exitCode=0
Sep 18 22:02:32.491011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.490669    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l"
Sep 18 22:02:32.491011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.490717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" event={"ID":"55f55e16-783f-41b4-802f-a73027f60c2d","Type":"ContainerDied","Data":"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"}
Sep 18 22:02:32.491011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.490781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-mvh9l" event={"ID":"55f55e16-783f-41b4-802f-a73027f60c2d","Type":"ContainerDied","Data":"adfde758df04155062d85606d196211c136c023a67779bd2e356aee5a190610f"}
Sep 18 22:02:32.491011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.490813    2563 scope.go:117] "RemoveContainer" containerID="ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"
Sep 18 22:02:32.491946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.491926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj" event={"ID":"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6","Type":"ContainerStarted","Data":"812fe3df4fbb6b5a033e927dd70d9c133a77ec2a5d218645a1835a5f5c606fc8"}
Sep 18 22:02:32.498813 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.498798    2563 scope.go:117] "RemoveContainer" containerID="ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"
Sep 18 22:02:32.499087 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:02:32.499065    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4\": container with ID starting with ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4 not found: ID does not exist" containerID="ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"
Sep 18 22:02:32.499164 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.499093    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4"} err="failed to get container status \"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4\": rpc error: code = NotFound desc = could not find container \"ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4\": container with ID starting with ebadd2f4f21acd4ff0fc4eab900a954c75180caec94a6904c56442ddc3296ae4 not found: ID does not exist"
Sep 18 22:02:32.515636 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.515614    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:02:32.533580 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:32.533556    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-575575dfd7-mvh9l"]
Sep 18 22:02:33.498576 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:33.498540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj" event={"ID":"6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6","Type":"ContainerStarted","Data":"a6db4f079b5ddf97392ba018e40e832d833f7c7fdab70cd7ff019cf190503471"}
Sep 18 22:02:33.499045 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:33.498670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:02:33.881866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:02:33.881840    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f55e16-783f-41b4-802f-a73027f60c2d" path="/var/lib/kubelet/pods/55f55e16-783f-41b4-802f-a73027f60c2d/volumes"
Sep 18 22:03:04.509095 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:04.509057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj"
Sep 18 22:03:04.529253 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:04.529197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575575dfd7-nrvzj" podStartSLOduration=33.133970356 podStartE2EDuration="33.529182704s" totalImagesPullingTime="395.212348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:02:33.575626706 +0000 UTC m=+693.774335005" watchObservedRunningTime="2026-09-18 22:03:04.529182704 +0000 UTC m=+724.727891003"
Sep 18 22:03:05.909854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.909825    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-x7szn"]
Sep 18 22:03:05.910201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.910085    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f55e16-783f-41b4-802f-a73027f60c2d" containerName="manager"
Sep 18 22:03:05.910201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.910096    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f55e16-783f-41b4-802f-a73027f60c2d" containerName="manager"
Sep 18 22:03:05.910201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.910157    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f55e16-783f-41b4-802f-a73027f60c2d" containerName="manager"
Sep 18 22:03:05.914143 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.914126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:05.917344 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.917326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 22:03:05.917625 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.917608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-t8bzd\""
Sep 18 22:03:05.942135 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:05.942114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x7szn"]
Sep 18 22:03:06.066370 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.066341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkh6\" (UniqueName: \"kubernetes.io/projected/f85225eb-ede0-4de8-841c-4a0f31721cf5-kube-api-access-vzkh6\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.066514 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.066388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85225eb-ede0-4de8-841c-4a0f31721cf5-tls-certs\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.167093 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.167020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkh6\" (UniqueName: \"kubernetes.io/projected/f85225eb-ede0-4de8-841c-4a0f31721cf5-kube-api-access-vzkh6\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.167093 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.167056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85225eb-ede0-4de8-841c-4a0f31721cf5-tls-certs\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.169472 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.169451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85225eb-ede0-4de8-841c-4a0f31721cf5-tls-certs\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.177671 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.177646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkh6\" (UniqueName: \"kubernetes.io/projected/f85225eb-ede0-4de8-841c-4a0f31721cf5-kube-api-access-vzkh6\") pod \"model-serving-api-7b954ffc95-x7szn\" (UID: \"f85225eb-ede0-4de8-841c-4a0f31721cf5\") " pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.224498 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.224474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:06.512626 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.512589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-x7szn"]
Sep 18 22:03:06.515134 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:03:06.515107    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85225eb_ede0_4de8_841c_4a0f31721cf5.slice/crio-c2d2a1e5729e98adbc2b710bad84611b3e5d89362693a67d2f938421f2e01490 WatchSource:0}: Error finding container c2d2a1e5729e98adbc2b710bad84611b3e5d89362693a67d2f938421f2e01490: Status 404 returned error can't find the container with id c2d2a1e5729e98adbc2b710bad84611b3e5d89362693a67d2f938421f2e01490
Sep 18 22:03:06.642425 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:06.642384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x7szn" event={"ID":"f85225eb-ede0-4de8-841c-4a0f31721cf5","Type":"ContainerStarted","Data":"c2d2a1e5729e98adbc2b710bad84611b3e5d89362693a67d2f938421f2e01490"}
Sep 18 22:03:08.660267 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:08.660229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-x7szn" event={"ID":"f85225eb-ede0-4de8-841c-4a0f31721cf5","Type":"ContainerStarted","Data":"c1bd01a89affa668ac3715e322cb3c112ff08900076fb96cf3095a9a2b5c6b31"}
Sep 18 22:03:08.660666 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:08.660418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:03:08.683573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:08.683530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-x7szn" podStartSLOduration=2.205958749 podStartE2EDuration="3.683513339s" totalImagesPullingTime="1.47755459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:03:08.682410466 +0000 UTC m=+728.881118755" watchObservedRunningTime="2026-09-18 22:03:08.683513339 +0000 UTC m=+728.882221638"
Sep 18 22:03:19.667964 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:03:19.667930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-x7szn"
Sep 18 22:05:10.091004 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.090970    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:05:10.093825 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.093800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.097471 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.097446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 22:05:10.097471 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.097457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-whrzw\""
Sep 18 22:05:10.097659 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.097446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 22:05:10.097659 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.097494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 22:05:10.097659 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.097458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-px4tn\""
Sep 18 22:05:10.112837 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.112808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:05:10.166413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.166373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.166511 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.166423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9scs\" (UniqueName: \"kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.267752 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.267720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.267752 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.267757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9scs\" (UniqueName: \"kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.270185 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.270162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.280843 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.280820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9scs\" (UniqueName: \"kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.404465 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.404389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:10.678822 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.678603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:05:10.681906 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:05:10.681873    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb372b592_efdd_46d7_8fd8_974181ff1277.slice/crio-cef8c1390050a866bda3d1050f661d476a929e7b1832040769a54a6f3c04d199 WatchSource:0}: Error finding container cef8c1390050a866bda3d1050f661d476a929e7b1832040769a54a6f3c04d199: Status 404 returned error can't find the container with id cef8c1390050a866bda3d1050f661d476a929e7b1832040769a54a6f3c04d199
Sep 18 22:05:10.684173 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:10.684154    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:05:11.190047 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:11.190016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" event={"ID":"b372b592-efdd-46d7-8fd8-974181ff1277","Type":"ContainerStarted","Data":"cef8c1390050a866bda3d1050f661d476a929e7b1832040769a54a6f3c04d199"}
Sep 18 22:05:13.203631 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:13.203596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" event={"ID":"b372b592-efdd-46d7-8fd8-974181ff1277","Type":"ContainerStarted","Data":"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"}
Sep 18 22:05:13.203976 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:13.203682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:13.221808 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:13.221764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" podStartSLOduration=1.7403941349999998 podStartE2EDuration="3.221749711s" totalImagesPullingTime="1.481355576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:05:13.221318136 +0000 UTC m=+853.420026437" watchObservedRunningTime="2026-09-18 22:05:13.221749711 +0000 UTC m=+853.420458011"
Sep 18 22:05:21.612758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.612729    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:05:21.614728 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.614712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.618215 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.618189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 22:05:21.618339 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.618315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-kv92f\""
Sep 18 22:05:21.618561 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.618542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-lzkjp\""
Sep 18 22:05:21.618661 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.618612    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 22:05:21.618949 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.618930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 22:05:21.626421 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.626372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:05:21.758541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.758506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.758729 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.758574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkd7l\" (UniqueName: \"kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.859463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.859437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.859583 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.859481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkd7l\" (UniqueName: \"kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.861837 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.861815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.868004 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.867942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkd7l\" (UniqueName: \"kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l\") pod \"fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:21.926191 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:21.926171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:22.202951 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:22.202735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:05:22.205070 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:05:22.205045    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42165239_be92_42bb_ad2d_618b782fe810.slice/crio-0c3dcca6bce915a42d2757d5b1e7fff22ad25d8cd75392564d4050ecba65f0ae WatchSource:0}: Error finding container 0c3dcca6bce915a42d2757d5b1e7fff22ad25d8cd75392564d4050ecba65f0ae: Status 404 returned error can't find the container with id 0c3dcca6bce915a42d2757d5b1e7fff22ad25d8cd75392564d4050ecba65f0ae
Sep 18 22:05:22.241781 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:22.241750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" event={"ID":"42165239-be92-42bb-ad2d-618b782fe810","Type":"ContainerStarted","Data":"0c3dcca6bce915a42d2757d5b1e7fff22ad25d8cd75392564d4050ecba65f0ae"}
Sep 18 22:05:23.249135 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:23.249094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" event={"ID":"42165239-be92-42bb-ad2d-618b782fe810","Type":"ContainerStarted","Data":"0a0c0488139fa3934eb4ca8672f0548c5dedecc85be1a6f9f85e0a313f5f096c"}
Sep 18 22:05:23.249581 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:23.249276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:05:23.269870 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:23.269827    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podStartSLOduration=2.26981229 podStartE2EDuration="2.26981229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:05:23.268267625 +0000 UTC m=+863.466975935" watchObservedRunningTime="2026-09-18 22:05:23.26981229 +0000 UTC m=+863.468520588"
Sep 18 22:05:44.209363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:44.209335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:05:53.041799 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:53.041713    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:05:54.209931 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:54.209890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:05:54.255410 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:05:54.255368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:06:01.359989 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:01.359951    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:06:01.861206 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:01.861178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:06:01.861983 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:01.861959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:06:04.209667 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.209630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:04.254904 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.254876    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:04.455126 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.455082    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 18 22:06:04.455239 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:06:04.455126    2563 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main" containerID="cri-o://08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"
Sep 18 22:06:04.455239 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.455173    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main" containerID="cri-o://08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6" gracePeriod=49
Sep 18 22:06:04.586388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.586368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv_b372b592-efdd-46d7-8fd8-974181ff1277/main/0.log"
Sep 18 22:06:04.587552 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.587536    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:06:04.678709 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.678687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9scs\" (UniqueName: \"kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs\") pod \"b372b592-efdd-46d7-8fd8-974181ff1277\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") "
Sep 18 22:06:04.678826 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.678749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs\") pod \"b372b592-efdd-46d7-8fd8-974181ff1277\" (UID: \"b372b592-efdd-46d7-8fd8-974181ff1277\") "
Sep 18 22:06:04.680809 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.680780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs" (OuterVolumeSpecName: "kube-api-access-t9scs") pod "b372b592-efdd-46d7-8fd8-974181ff1277" (UID: "b372b592-efdd-46d7-8fd8-974181ff1277"). InnerVolumeSpecName "kube-api-access-t9scs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:06:04.680809 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.680797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b372b592-efdd-46d7-8fd8-974181ff1277" (UID: "b372b592-efdd-46d7-8fd8-974181ff1277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:06:04.779914 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.779861    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9scs\" (UniqueName: \"kubernetes.io/projected/b372b592-efdd-46d7-8fd8-974181ff1277-kube-api-access-t9scs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:06:04.779914 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:04.779882    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b372b592-efdd-46d7-8fd8-974181ff1277-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:06:05.437174 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437148    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv_b372b592-efdd-46d7-8fd8-974181ff1277/main/0.log"
Sep 18 22:06:05.437763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437190    2563 generic.go:356] "Generic (PLEG): container finished" podID="b372b592-efdd-46d7-8fd8-974181ff1277" containerID="08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6" exitCode=1
Sep 18 22:06:05.437763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" event={"ID":"b372b592-efdd-46d7-8fd8-974181ff1277","Type":"ContainerDied","Data":"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"}
Sep 18 22:06:05.437763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437258    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"
Sep 18 22:06:05.437763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437277    2563 scope.go:117] "RemoveContainer" containerID="08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"
Sep 18 22:06:05.437763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.437268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv" event={"ID":"b372b592-efdd-46d7-8fd8-974181ff1277","Type":"ContainerDied","Data":"cef8c1390050a866bda3d1050f661d476a929e7b1832040769a54a6f3c04d199"}
Sep 18 22:06:05.446454 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.446435    2563 scope.go:117] "RemoveContainer" containerID="08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"
Sep 18 22:06:05.446734 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:06:05.446717    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6\": container with ID starting with 08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6 not found: ID does not exist" containerID="08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"
Sep 18 22:06:05.446796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.446743    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6"} err="failed to get container status \"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6\": rpc error: code = NotFound desc = could not find container \"08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6\": container with ID starting with 08180cf8a320856c769f644f3ae1e69646a166a632207f0d0acb187c1badfba6 not found: ID does not exist"
Sep 18 22:06:05.460121 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.460097    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:06:05.463517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.463495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5b4c758rsxqv"]
Sep 18 22:06:05.883364 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:05.883335    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" path="/var/lib/kubelet/pods/b372b592-efdd-46d7-8fd8-974181ff1277/volumes"
Sep 18 22:06:14.255716 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:14.255672    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:16.380590 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:16.380548    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" containerID="cri-o://0a0c0488139fa3934eb4ca8672f0548c5dedecc85be1a6f9f85e0a313f5f096c" gracePeriod=45
Sep 18 22:06:24.254818 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:24.254780    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:24.255187 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:24.254891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:06:34.254842 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:34.254806    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:44.255631 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:44.255599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:06:46.617561 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.617535    2563 generic.go:356] "Generic (PLEG): container finished" podID="42165239-be92-42bb-ad2d-618b782fe810" containerID="0a0c0488139fa3934eb4ca8672f0548c5dedecc85be1a6f9f85e0a313f5f096c" exitCode=0
Sep 18 22:06:46.617878 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.617582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" event={"ID":"42165239-be92-42bb-ad2d-618b782fe810","Type":"ContainerDied","Data":"0a0c0488139fa3934eb4ca8672f0548c5dedecc85be1a6f9f85e0a313f5f096c"}
Sep 18 22:06:46.633094 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.633074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:06:46.670864 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.670771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs\") pod \"42165239-be92-42bb-ad2d-618b782fe810\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") "
Sep 18 22:06:46.671011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.670906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkd7l\" (UniqueName: \"kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l\") pod \"42165239-be92-42bb-ad2d-618b782fe810\" (UID: \"42165239-be92-42bb-ad2d-618b782fe810\") "
Sep 18 22:06:46.673203 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.673179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42165239-be92-42bb-ad2d-618b782fe810" (UID: "42165239-be92-42bb-ad2d-618b782fe810"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:06:46.673432 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.673390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l" (OuterVolumeSpecName: "kube-api-access-nkd7l") pod "42165239-be92-42bb-ad2d-618b782fe810" (UID: "42165239-be92-42bb-ad2d-618b782fe810"). InnerVolumeSpecName "kube-api-access-nkd7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:06:46.772038 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.772012    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42165239-be92-42bb-ad2d-618b782fe810-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:06:46.772038 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:46.772037    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkd7l\" (UniqueName: \"kubernetes.io/projected/42165239-be92-42bb-ad2d-618b782fe810-kube-api-access-nkd7l\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:06:47.623128 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.623096    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"
Sep 18 22:06:47.623713 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.623097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt" event={"ID":"42165239-be92-42bb-ad2d-618b782fe810","Type":"ContainerDied","Data":"0c3dcca6bce915a42d2757d5b1e7fff22ad25d8cd75392564d4050ecba65f0ae"}
Sep 18 22:06:47.623713 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.623247    2563 scope.go:117] "RemoveContainer" containerID="0a0c0488139fa3934eb4ca8672f0548c5dedecc85be1a6f9f85e0a313f5f096c"
Sep 18 22:06:47.646959 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.646934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:06:47.650936 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.650906    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-67d954b65c-bxldt"]
Sep 18 22:06:47.882617 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:47.882541    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42165239-be92-42bb-ad2d-618b782fe810" path="/var/lib/kubelet/pods/42165239-be92-42bb-ad2d-618b782fe810/volumes"
Sep 18 22:06:49.923158 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923123    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:06:49.923616 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main"
Sep 18 22:06:49.923616 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main"
Sep 18 22:06:49.923616 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main"
Sep 18 22:06:49.923616 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main"
Sep 18 22:06:49.923818 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923719    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b372b592-efdd-46d7-8fd8-974181ff1277" containerName="main"
Sep 18 22:06:49.923818 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.923733    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42165239-be92-42bb-ad2d-618b782fe810" containerName="main"
Sep 18 22:06:49.926684 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.926662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:49.929110 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.929090    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 22:06:49.929911 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.929893    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 22:06:49.930192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.930172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 22:06:49.930259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.930210    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-cnsnk\""
Sep 18 22:06:49.930259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.930224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-q6ddn\""
Sep 18 22:06:49.937012 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.936990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:06:49.997097 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.997071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:49.997212 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.997103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqrq\" (UniqueName: \"kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:49.997212 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.997126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:49.997212 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:49.997143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.098528 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.098495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.098528 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.098530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.098774 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.098594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.098774 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.098636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqrq\" (UniqueName: \"kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.098900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.098879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.099042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.099006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.101149 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.101126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.106762 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.106739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqrq\" (UniqueName: \"kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.237624 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.237529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.526000 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.525941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:06:50.527188 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:06:50.527156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152b3270_a723_43b5_a56e_4bac90a48c66.slice/crio-dec9389754ec7aa416aef8ddc7d15fa0eaef31762d66f679d1b390b45bad3fe0 WatchSource:0}: Error finding container dec9389754ec7aa416aef8ddc7d15fa0eaef31762d66f679d1b390b45bad3fe0: Status 404 returned error can't find the container with id dec9389754ec7aa416aef8ddc7d15fa0eaef31762d66f679d1b390b45bad3fe0
Sep 18 22:06:50.640603 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.640567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" event={"ID":"152b3270-a723-43b5-a56e-4bac90a48c66","Type":"ContainerStarted","Data":"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"}
Sep 18 22:06:50.640794 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.640649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" event={"ID":"152b3270-a723-43b5-a56e-4bac90a48c66","Type":"ContainerStarted","Data":"dec9389754ec7aa416aef8ddc7d15fa0eaef31762d66f679d1b390b45bad3fe0"}
Sep 18 22:06:50.640794 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.640676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:06:50.661903 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:06:50.661835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podStartSLOduration=1.661822149 podStartE2EDuration="1.661822149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:06:50.661049903 +0000 UTC m=+950.859758203" watchObservedRunningTime="2026-09-18 22:06:50.661822149 +0000 UTC m=+950.860530448"
Sep 18 22:07:21.648696 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:07:21.648622    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:07:40.532611 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:07:40.532578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:07:41.648828 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:07:41.648793    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:07:51.648214 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:07:51.648169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:07:55.551974 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:07:55.551931    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" containerID="cri-o://246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf" gracePeriod=45
Sep 18 22:08:01.648819 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:01.648780    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:08:01.649197 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:01.648876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:08:11.648941 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:11.648901    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:08:21.648846 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:21.648807    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:08:25.787742 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.787720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:08:25.876704 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.876674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache\") pod \"152b3270-a723-43b5-a56e-4bac90a48c66\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") "
Sep 18 22:08:25.876833 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.876744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs\") pod \"152b3270-a723-43b5-a56e-4bac90a48c66\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") "
Sep 18 22:08:25.876833 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.876778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache\") pod \"152b3270-a723-43b5-a56e-4bac90a48c66\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") "
Sep 18 22:08:25.876919 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.876845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dqrq\" (UniqueName: \"kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq\") pod \"152b3270-a723-43b5-a56e-4bac90a48c66\" (UID: \"152b3270-a723-43b5-a56e-4bac90a48c66\") "
Sep 18 22:08:25.877079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.876986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "152b3270-a723-43b5-a56e-4bac90a48c66" (UID: "152b3270-a723-43b5-a56e-4bac90a48c66"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:08:25.877079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.877050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "152b3270-a723-43b5-a56e-4bac90a48c66" (UID: "152b3270-a723-43b5-a56e-4bac90a48c66"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:08:25.877241 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.877112    2563 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-cachi2-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:08:25.877241 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.877130    2563 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/152b3270-a723-43b5-a56e-4bac90a48c66-tokenizer-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:08:25.878987 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.878963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "152b3270-a723-43b5-a56e-4bac90a48c66" (UID: "152b3270-a723-43b5-a56e-4bac90a48c66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:08:25.879142 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.879119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq" (OuterVolumeSpecName: "kube-api-access-6dqrq") pod "152b3270-a723-43b5-a56e-4bac90a48c66" (UID: "152b3270-a723-43b5-a56e-4bac90a48c66"). InnerVolumeSpecName "kube-api-access-6dqrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:08:25.978458 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.978436    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqrq\" (UniqueName: \"kubernetes.io/projected/152b3270-a723-43b5-a56e-4bac90a48c66-kube-api-access-6dqrq\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:08:25.978458 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:25.978457    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152b3270-a723-43b5-a56e-4bac90a48c66-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:08:26.054651 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.054618    2563 generic.go:356] "Generic (PLEG): container finished" podID="152b3270-a723-43b5-a56e-4bac90a48c66" containerID="246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf" exitCode=0
Sep 18 22:08:26.054796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.054664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"
Sep 18 22:08:26.054796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.054701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" event={"ID":"152b3270-a723-43b5-a56e-4bac90a48c66","Type":"ContainerDied","Data":"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"}
Sep 18 22:08:26.054796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.054750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t" event={"ID":"152b3270-a723-43b5-a56e-4bac90a48c66","Type":"ContainerDied","Data":"dec9389754ec7aa416aef8ddc7d15fa0eaef31762d66f679d1b390b45bad3fe0"}
Sep 18 22:08:26.054796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.054773    2563 scope.go:117] "RemoveContainer" containerID="246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"
Sep 18 22:08:26.063033 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.063018    2563 scope.go:117] "RemoveContainer" containerID="246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"
Sep 18 22:08:26.063286 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:08:26.063267    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf\": container with ID starting with 246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf not found: ID does not exist" containerID="246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"
Sep 18 22:08:26.063352 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.063305    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf"} err="failed to get container status \"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf\": rpc error: code = NotFound desc = could not find container \"246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf\": container with ID starting with 246199f5fa7a56055d113f8d0e17518f04e80cc0f66a19a3cfbb89844f0f32bf not found: ID does not exist"
Sep 18 22:08:26.072116 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.072091    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:08:26.076337 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:26.076316    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-c5fbch5t"]
Sep 18 22:08:27.882361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:27.882327    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" path="/var/lib/kubelet/pods/152b3270-a723-43b5-a56e-4bac90a48c66/volumes"
Sep 18 22:08:33.789766 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.789732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"]
Sep 18 22:08:33.790239 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.790154    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main"
Sep 18 22:08:33.790239 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.790171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main"
Sep 18 22:08:33.790349 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.790246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="152b3270-a723-43b5-a56e-4bac90a48c66" containerName="main"
Sep 18 22:08:33.794729 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.794705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.798074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.798047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 22:08:33.798214 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.798111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-d6hkm\""
Sep 18 22:08:33.798214 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.798176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 22:08:33.798320 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.798270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 22:08:33.802855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.802828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"]
Sep 18 22:08:33.943226 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/5d32db0e-866f-480a-85a4-0327717bc9af-kube-api-access-bk2lw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.943379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.943379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.943379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d32db0e-866f-480a-85a4-0327717bc9af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.943379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.943644 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.943390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:33.986887 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.986858    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"]
Sep 18 22:08:33.990114 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.990092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:33.992409 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:33.992376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq\""
Sep 18 22:08:34.000668 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.000648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"]
Sep 18 22:08:34.044155 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044155 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044155 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d32db0e-866f-480a-85a4-0327717bc9af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044340 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044340 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044340 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/5d32db0e-866f-480a-85a4-0327717bc9af-kube-api-access-bk2lw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044528 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044596 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.044693 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.044669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.046509 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.046486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d32db0e-866f-480a-85a4-0327717bc9af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.046770 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.046753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d32db0e-866f-480a-85a4-0327717bc9af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.052391 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.052369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/5d32db0e-866f-480a-85a4-0327717bc9af-kube-api-access-bk2lw\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk\" (UID: \"5d32db0e-866f-480a-85a4-0327717bc9af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.105519 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.105487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:34.145384 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.145344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7vp\" (UniqueName: \"kubernetes.io/projected/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-kube-api-access-2s7vp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.145540 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.145490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.245946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.245902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7vp\" (UniqueName: \"kubernetes.io/projected/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-kube-api-access-2s7vp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.246111 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.246010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.248675 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.248649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.254587 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.254565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7vp\" (UniqueName: \"kubernetes.io/projected/b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa-kube-api-access-2s7vp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp\" (UID: \"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.301550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.301496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:34.382513 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.382425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"]
Sep 18 22:08:34.384810 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:08:34.384784    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d32db0e_866f_480a_85a4_0327717bc9af.slice/crio-27e07a9294d5299d11af889314381294ca0ac3ad97fbf98faf2818f4a5d54a60 WatchSource:0}: Error finding container 27e07a9294d5299d11af889314381294ca0ac3ad97fbf98faf2818f4a5d54a60: Status 404 returned error can't find the container with id 27e07a9294d5299d11af889314381294ca0ac3ad97fbf98faf2818f4a5d54a60
Sep 18 22:08:34.579053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:34.578984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"]
Sep 18 22:08:34.582681 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:08:34.582641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cc8ea0_9ab3_4a08_8ce0_8a5d663ab9fa.slice/crio-241e1784fed91e0177fbb3badbc32ec1ca6cdddf3f18396a012fc0f316ea4a08 WatchSource:0}: Error finding container 241e1784fed91e0177fbb3badbc32ec1ca6cdddf3f18396a012fc0f316ea4a08: Status 404 returned error can't find the container with id 241e1784fed91e0177fbb3badbc32ec1ca6cdddf3f18396a012fc0f316ea4a08
Sep 18 22:08:35.101847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.101768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk" event={"ID":"5d32db0e-866f-480a-85a4-0327717bc9af","Type":"ContainerStarted","Data":"27e07a9294d5299d11af889314381294ca0ac3ad97fbf98faf2818f4a5d54a60"}
Sep 18 22:08:35.104116 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.104095    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/0.log"
Sep 18 22:08:35.104259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.104139    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="babf35b1ce822482a31bd1c4d719e3cfcb78c626f14eb9a0466ca6b5efe9383a" exitCode=1
Sep 18 22:08:35.104259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.104238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"babf35b1ce822482a31bd1c4d719e3cfcb78c626f14eb9a0466ca6b5efe9383a"}
Sep 18 22:08:35.104366 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.104282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerStarted","Data":"241e1784fed91e0177fbb3badbc32ec1ca6cdddf3f18396a012fc0f316ea4a08"}
Sep 18 22:08:35.104507 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:35.104477    2563 scope.go:117] "RemoveContainer" containerID="babf35b1ce822482a31bd1c4d719e3cfcb78c626f14eb9a0466ca6b5efe9383a"
Sep 18 22:08:36.110047 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.110013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk" event={"ID":"5d32db0e-866f-480a-85a4-0327717bc9af","Type":"ContainerStarted","Data":"87f4cc7939fe9d118b854ecd7a84c1fc1648d25890a485a6080388341937fa51"}
Sep 18 22:08:36.111630 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.111612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/1.log"
Sep 18 22:08:36.112051 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.112036    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/0.log"
Sep 18 22:08:36.112119 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.112070    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c" exitCode=1
Sep 18 22:08:36.112160 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.112123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"}
Sep 18 22:08:36.112206 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.112163    2563 scope.go:117] "RemoveContainer" containerID="babf35b1ce822482a31bd1c4d719e3cfcb78c626f14eb9a0466ca6b5efe9383a"
Sep 18 22:08:36.112317 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.112301    2563 scope.go:117] "RemoveContainer" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"
Sep 18 22:08:36.112552 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:08:36.112526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:08:36.135761 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:36.133513    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk" podStartSLOduration=1.536830445 podStartE2EDuration="3.133495294s" totalImagesPullingTime="1.596664849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:08:36.132771007 +0000 UTC m=+1056.331479296" watchObservedRunningTime="2026-09-18 22:08:36.133495294 +0000 UTC m=+1056.332203598"
Sep 18 22:08:37.106151 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:37.106111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:37.117339 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:37.117317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/1.log"
Sep 18 22:08:37.117736 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:37.117650    2563 scope.go:117] "RemoveContainer" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"
Sep 18 22:08:37.117824 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:08:37.117806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:08:44.106469 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.106431    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:44.120278 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.120255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:44.158883 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.158861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk"
Sep 18 22:08:44.302032 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.302006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:44.302184 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.302045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:08:44.302379 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:44.302362    2563 scope.go:117] "RemoveContainer" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"
Sep 18 22:08:44.302596 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:08:44.302575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:08:58.877093 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:58.877067    2563 scope.go:117] "RemoveContainer" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"
Sep 18 22:08:59.217363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.217306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/2.log"
Sep 18 22:08:59.217751 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.217733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/1.log"
Sep 18 22:08:59.217841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.217773    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad" exitCode=1
Sep 18 22:08:59.217884 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.217847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"}
Sep 18 22:08:59.217920 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.217907    2563 scope.go:117] "RemoveContainer" containerID="6ec7b6ec02635b2a964457e2eeb5e70b1c43c419ec6448ae28e1cce7a0d7833c"
Sep 18 22:08:59.218260 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:08:59.218237    2563 scope.go:117] "RemoveContainer" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"
Sep 18 22:08:59.218453 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:08:59.218434    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:09:00.223462 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:00.223435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/2.log"
Sep 18 22:09:04.302052 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:04.302023    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:09:04.302445 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:04.302071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:09:04.302445 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:04.302334    2563 scope.go:117] "RemoveContainer" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"
Sep 18 22:09:04.302572 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:09:04.302531    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:09:17.876968 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:17.876929    2563 scope.go:117] "RemoveContainer" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"
Sep 18 22:09:17.877419 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:09:17.877164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:09:28.877265 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:28.877234    2563 scope.go:117] "RemoveContainer" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"
Sep 18 22:09:29.348831 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.348807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/3.log"
Sep 18 22:09:29.349254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.349237    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/2.log"
Sep 18 22:09:29.349361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.349277    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2" exitCode=1
Sep 18 22:09:29.349361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.349337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"}
Sep 18 22:09:29.349499 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.349413    2563 scope.go:117] "RemoveContainer" containerID="10fb224f44683cc3ed1da07eefb18d9f6f49adb1222cb410a79cb6d7a6428cad"
Sep 18 22:09:29.349699 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:29.349683    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:09:29.349883 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:09:29.349864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:09:30.355307 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:30.355279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/3.log"
Sep 18 22:09:34.302055 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:34.302006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:09:34.302485 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:34.302086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:09:34.302485 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:34.302354    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:09:34.302598 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:09:34.302535    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:09:49.876726 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:09:49.876693    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:09:49.877116 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:09:49.876894    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:10:00.876443 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:00.876391    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:10:00.876883 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:10:00.876594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:10:13.876905 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:13.876871    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:10:14.544734 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.544710    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:10:14.545146 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.545132    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/3.log"
Sep 18 22:10:14.545195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.545163    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06" exitCode=1
Sep 18 22:10:14.545230 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.545203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"}
Sep 18 22:10:14.545264 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.545245    2563 scope.go:117] "RemoveContainer" containerID="4dcd9dd3e58f48ba2a50dbe185e5ddd022114b8f8541321f2bcb4ac9ef4dbdf2"
Sep 18 22:10:14.545547 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:14.545526    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:10:14.545749 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:10:14.545726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:10:15.551358 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:15.551332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:10:24.302592 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:24.302540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:10:24.303089 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:24.302626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:10:24.303089 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:24.303042    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:10:24.303299 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:10:24.303275    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:10:38.877089 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:38.877060    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:10:38.877433 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:10:38.877229    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:10:52.877328 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:10:52.877296    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:10:52.877799 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:10:52.877488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:01.875292 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:01.875263    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:11:01.879271 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:01.879243    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:11:01.890122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:01.890102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:11:01.891593 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:01.891571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:11:05.877312 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:05.877282    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:11:05.877707 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:05.877475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:16.876475 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:16.876444    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:11:16.876871 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:16.876630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:29.876971 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:29.876935    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:11:29.877589 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:29.877170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:44.876306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:44.876238    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:11:44.963575 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:44.963549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:11:44.963695 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:44.963611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerStarted","Data":"fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"}
Sep 18 22:11:44.963855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:44.963833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:11:44.982971 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:44.982933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podStartSLOduration=191.982922661 podStartE2EDuration="3m11.982922661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:11:44.981780562 +0000 UTC m=+1245.180488862" watchObservedRunningTime="2026-09-18 22:11:44.982922661 +0000 UTC m=+1245.181630959"
Sep 18 22:11:45.969568 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.969544    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/5.log"
Sep 18 22:11:45.969962 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.969950    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/4.log"
Sep 18 22:11:45.970008 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.969982    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e" exitCode=1
Sep 18 22:11:45.970052 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.970038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"}
Sep 18 22:11:45.970094 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.970084    2563 scope.go:117] "RemoveContainer" containerID="4a8219473ae5ac4df0cb0a469cec942ee9711ff66bc55079380ee708a215fd06"
Sep 18 22:11:45.970266 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:45.970248    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:11:45.970501 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:45.970476    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:46.977041 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:46.977012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/5.log"
Sep 18 22:11:46.977500 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:46.977369    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:11:46.977598 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:46.977574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:11:54.302199 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:54.302166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:11:54.302590 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:11:54.302494    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:11:54.302675 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:11:54.302655    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:12:07.876550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:12:07.876502    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:12:07.876978 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:12:07.876769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:12:19.876405 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:12:19.876358    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:12:19.876887 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:12:19.876558    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:12:32.876878 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:12:32.876845    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:12:32.877339 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:12:32.877015    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:12:45.877228 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:12:45.877196    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:12:45.877633 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:12:45.877365    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:13:00.876855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:13:00.876826    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:13:00.877268 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:13:00.876991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:13:12.876749 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:13:12.876721    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:13:12.877190 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:13:12.876888    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:13:24.877005 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:13:24.876889    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:13:24.877473 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:13:24.877165    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:13:36.876780 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:13:36.876747    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:13:36.877151 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:13:36.876917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:13:51.878883 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:13:51.878853    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:13:51.879331 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:13:51.879044    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:02.877295 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:02.877265    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:14:02.877681 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:02.877470    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:13.876811 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:13.876783    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:14:13.877196 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:13.876957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:25.876360 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:25.876335    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:14:26.691625 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.691598    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/6.log"
Sep 18 22:14:26.692036 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.692020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/5.log"
Sep 18 22:14:26.692096 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.692053    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34" exitCode=1
Sep 18 22:14:26.692149 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.692126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"}
Sep 18 22:14:26.692197 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.692187    2563 scope.go:117] "RemoveContainer" containerID="fc13b1fbdb8fff484cf437e6dfd6347a2d38b6f0e307482043de30e3fad3664e"
Sep 18 22:14:26.692460 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:26.692446    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:14:26.692646 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:26.692629    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:27.697539 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:27.697510    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/6.log"
Sep 18 22:14:34.301633 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:34.301601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:14:34.302084 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:34.301930    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:14:34.302131 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:34.302095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:34.964288 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:34.964261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:14:34.964639 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:34.964626    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:14:34.964806 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:34.964787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:14:48.876982 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:14:48.876900    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:14:48.877436 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:14:48.877108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:15:01.879053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:15:01.879025    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:15:01.879430 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:15:01.879187    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:15:15.876424 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:15:15.876379    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:15:15.876990 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:15:15.876590    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:15:28.876315 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:15:28.876281    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:15:28.876812 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:15:28.876479    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:15:39.876716 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:15:39.876684    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:15:39.877156 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:15:39.876893    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:15:51.886001 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:15:51.885970    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:15:51.886463 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:15:51.886109    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:16:01.901868 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:01.901843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/6.log"
Sep 18 22:16:01.905235 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:01.905206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/6.log"
Sep 18 22:16:01.918905 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:01.918883    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:16:01.922808 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:01.922789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:16:05.876557 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:05.876528    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:16:05.876915 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:16:05.876694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:16:16.876646 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:16.876575    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:16:16.877074 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:16:16.876740    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:16:28.876727 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:28.876694    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:16:28.877093 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:16:28.876857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:16:42.877247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:42.877208    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:16:42.877730 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:16:42.877469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:16:57.876885 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:16:57.876852    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:16:57.879135 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:16:57.877021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:17:09.876747 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:17:09.876712    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:17:09.877130 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:17:09.876872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:17:21.878663 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:17:21.878631    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:17:21.879120 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:17:21.878789    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:17:34.876800 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:17:34.876770    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:17:34.877229 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:17:34.876929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:17:47.877089 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:17:47.877000    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:17:47.877562 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:17:47.877228    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:18:02.877229 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:18:02.877199    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:18:02.877651 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:18:02.877355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:18:16.876490 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:18:16.876459    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:18:16.876949 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:18:16.876657    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:18:29.876695 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:18:29.876668    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:18:29.877124 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:18:29.876824    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:18:44.876987 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:18:44.876957    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:18:44.877370 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:18:44.877118    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:18:57.876695 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:18:57.876664    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:18:57.877077 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:18:57.876837    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:12.876465 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:12.876433    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:19:12.876925 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:12.876607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:24.877006 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:24.876927    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:19:24.877506 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:24.877115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:36.876271 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:36.876241    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:19:37.049506 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.049485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:19:37.049910 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.049896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/6.log"
Sep 18 22:19:37.049957 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.049928    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6" exitCode=1
Sep 18 22:19:37.049994 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.049981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"}
Sep 18 22:19:37.050040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.050027    2563 scope.go:117] "RemoveContainer" containerID="38d9fa33cd55c689eacb9b57d85a182470d6189d93971ca63432b45eeb754f34"
Sep 18 22:19:37.050305 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:37.050284    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:19:37.050537 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:37.050513    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:38.060880 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:38.060849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:19:44.302494 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:44.302462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:19:44.302932 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:44.302776    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:19:44.302978 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:44.302949    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:44.964171 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:44.964141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:19:45.100544 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:45.100518    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:19:45.100699 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:45.100670    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:19:56.877157 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:19:56.877121    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:19:56.877647 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:19:56.877345    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:20:07.877262 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:20:07.877229    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:20:07.877760 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:20:07.877386    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:20:18.876770 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:20:18.876741    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:20:18.877195 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:20:18.876923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:20:32.876658 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:20:32.876622    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:20:32.877140 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:20:32.876808    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:20:45.877267 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:20:45.877185    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:20:45.877722 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:20:45.877474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:20:57.882411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:20:57.882367    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:20:57.882811 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:20:57.882555    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:21:01.936343 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:01.936310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:21:01.940485 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:01.940463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:21:01.947488 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:01.947462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:21:01.951290 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:01.951275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:21:11.878562 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:11.878529    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:21:11.878942 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:21:11.878722    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:21:23.876769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:23.876738    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:21:23.877159 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:21:23.876895    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:21:38.877209 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:38.877179    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:21:38.877681 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:21:38.877372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:21:52.876299 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:21:52.876263    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:21:52.877014 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:21:52.876704    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:22:07.877220 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:22:07.877192    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:22:07.877649 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:22:07.877349    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:22:18.876866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:22:18.876783    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:22:18.877340 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:22:18.876969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:22:30.877275 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:22:30.877244    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:22:30.877759 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:22:30.877435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:22:42.877227 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:22:42.877198    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:22:42.877654 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:22:42.877351    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:22:54.877573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:22:54.877542    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:22:54.877947 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:22:54.877746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:23:09.876720 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:09.876689    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:23:09.877262 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:23:09.876865    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:23:23.725101 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.725070    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:23:23.728324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.728302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.732523 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.732494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 22:23:23.732638 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.732605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-crnmt\""
Sep 18 22:23:23.732638 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.732617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 22:23:23.732753 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.732609    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 22:23:23.732753 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.732724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-q5sc9\""
Sep 18 22:23:23.742332 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.742310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:23:23.759237 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.759212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.759367 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.759263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2n9c\" (UniqueName: \"kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.859821 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.859793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2n9c\" (UniqueName: \"kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.859929 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.859849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.862490 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.862465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:23.867862 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:23.867836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2n9c\" (UniqueName: \"kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c\") pod \"auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:24.038587 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:24.038502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:24.316592 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:24.316567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:23:24.319090 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:23:24.319056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7e636a_b7f3_431f_9fd3_426a6e1954c9.slice/crio-5f381c7f7a9fe0bc790cd557308cfdf1bee0643b4305a2e0856d0a6a30e8fbf9 WatchSource:0}: Error finding container 5f381c7f7a9fe0bc790cd557308cfdf1bee0643b4305a2e0856d0a6a30e8fbf9: Status 404 returned error can't find the container with id 5f381c7f7a9fe0bc790cd557308cfdf1bee0643b4305a2e0856d0a6a30e8fbf9
Sep 18 22:23:24.876509 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:24.876471    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:23:24.876859 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:23:24.876624    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:23:25.059005 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:25.058972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" event={"ID":"dd7e636a-b7f3-431f-9fd3-426a6e1954c9","Type":"ContainerStarted","Data":"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"}
Sep 18 22:23:25.059153 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:25.059021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" event={"ID":"dd7e636a-b7f3-431f-9fd3-426a6e1954c9","Type":"ContainerStarted","Data":"5f381c7f7a9fe0bc790cd557308cfdf1bee0643b4305a2e0856d0a6a30e8fbf9"}
Sep 18 22:23:25.059153 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:25.059051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:23:25.091116 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:25.091068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podStartSLOduration=2.09105245 podStartE2EDuration="2.09105245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:25.088514946 +0000 UTC m=+1945.287223227" watchObservedRunningTime="2026-09-18 22:23:25.09105245 +0000 UTC m=+1945.289760748"
Sep 18 22:23:29.135569 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:29.135539    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk_5d32db0e-866f-480a-85a4-0327717bc9af/main/0.log"
Sep 18 22:23:29.142501 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:29.142475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:23:29.148793 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:29.148773    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:23:39.877312 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:39.877277    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:23:39.877772 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:23:39.877483    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:23:52.876459 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:52.876372    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:23:52.876888 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:23:52.876659    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:23:56.065591 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:23:56.065558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:24:05.877080 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:05.877047    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:24:05.877548 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:05.877268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:17.877309 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:17.877274    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:24:17.877770 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:17.877506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:30.876620 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:30.876589    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:24:30.877013 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:30.876787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:42.876486 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:42.876455    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:24:43.401519 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.401497    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/8.log"
Sep 18 22:24:43.401956 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.401937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/7.log"
Sep 18 22:24:43.402072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.401977    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5" exitCode=1
Sep 18 22:24:43.402072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.402048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"}
Sep 18 22:24:43.402173 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.402106    2563 scope.go:117] "RemoveContainer" containerID="dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6"
Sep 18 22:24:43.402411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:43.402373    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:24:43.402607 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:43.402587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:44.302553 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:44.302527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:24:44.408070 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:44.408044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/8.log"
Sep 18 22:24:44.408369 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:44.408351    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:24:44.408571 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:44.408551    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:44.964235 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:44.964204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:24:45.413300 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:45.413275    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:24:45.413675 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:45.413480    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:24:55.876957 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:24:55.876926    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:24:55.877480 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:24:55.877130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:25:09.877281 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:25:09.877252    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:25:09.877664 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:25:09.877435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:25:21.879008 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:25:21.878924    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:25:21.879353 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:25:21.879138    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:25:34.877157 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:25:34.877127    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:25:34.877639 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:25:34.877294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:25:47.882769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:25:47.882739    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:25:47.883210 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:25:47.882893    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:25:59.052639 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:25:59.052599    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:26:01.959940 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:01.959915    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/8.log"
Sep 18 22:26:01.968564 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:01.968542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/8.log"
Sep 18 22:26:01.972048 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:01.972016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:26:01.980234 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:01.980216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:26:02.876686 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:02.876655    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:26:02.876859 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:02.876825    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:26:06.065927 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:06.065888    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:26:14.076791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:14.076742    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main" containerID="cri-o://ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7" gracePeriod=45
Sep 18 22:26:16.065986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:16.065951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:26:16.877048 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:16.877022    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:26:16.877211 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:16.877181    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:26:26.065131 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:26.065094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:26:26.065595 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:26.065213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:26:28.876841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:28.876809    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:26:28.877210 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:28.877006    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:26:36.065298 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:36.065261    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:26:41.878130 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:41.878097    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:26:41.878554 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:41.878277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:26:44.324949 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.324925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:26:44.413387 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.413318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs\") pod \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") "
Sep 18 22:26:44.413387 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.413382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2n9c\" (UniqueName: \"kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c\") pod \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\" (UID: \"dd7e636a-b7f3-431f-9fd3-426a6e1954c9\") "
Sep 18 22:26:44.415450 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.415425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c" (OuterVolumeSpecName: "kube-api-access-w2n9c") pod "dd7e636a-b7f3-431f-9fd3-426a6e1954c9" (UID: "dd7e636a-b7f3-431f-9fd3-426a6e1954c9"). InnerVolumeSpecName "kube-api-access-w2n9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:26:44.415545 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.415473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd7e636a-b7f3-431f-9fd3-426a6e1954c9" (UID: "dd7e636a-b7f3-431f-9fd3-426a6e1954c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:26:44.514068 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.514041    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:26:44.514068 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.514064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2n9c\" (UniqueName: \"kubernetes.io/projected/dd7e636a-b7f3-431f-9fd3-426a6e1954c9-kube-api-access-w2n9c\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:26:44.940257 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.940225    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerID="ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7" exitCode=0
Sep 18 22:26:44.940388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.940271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" event={"ID":"dd7e636a-b7f3-431f-9fd3-426a6e1954c9","Type":"ContainerDied","Data":"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"}
Sep 18 22:26:44.940388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.940275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"
Sep 18 22:26:44.940388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.940309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm" event={"ID":"dd7e636a-b7f3-431f-9fd3-426a6e1954c9","Type":"ContainerDied","Data":"5f381c7f7a9fe0bc790cd557308cfdf1bee0643b4305a2e0856d0a6a30e8fbf9"}
Sep 18 22:26:44.940388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.940339    2563 scope.go:117] "RemoveContainer" containerID="ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"
Sep 18 22:26:44.948521 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.948503    2563 scope.go:117] "RemoveContainer" containerID="ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"
Sep 18 22:26:44.948764 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:44.948745    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7\": container with ID starting with ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7 not found: ID does not exist" containerID="ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"
Sep 18 22:26:44.948819 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.948771    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7"} err="failed to get container status \"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7\": rpc error: code = NotFound desc = could not find container \"ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7\": container with ID starting with ccbb547bf63b92c3ae79158e943e982d54834fe2c85b63194f15526f1bdcadd7 not found: ID does not exist"
Sep 18 22:26:44.965169 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.965145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:26:44.970574 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:44.970553    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-65b65bdfc9-7qcxm"]
Sep 18 22:26:45.882313 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:45.882276    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" path="/var/lib/kubelet/pods/dd7e636a-b7f3-431f-9fd3-426a6e1954c9/volumes"
Sep 18 22:26:51.899587 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.899554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:26:51.900071 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.899830    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main"
Sep 18 22:26:51.900071 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.899841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main"
Sep 18 22:26:51.900071 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.899906    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7e636a-b7f3-431f-9fd3-426a6e1954c9" containerName="main"
Sep 18 22:26:51.904134 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.904110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:51.908635 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.908614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-lwbg7\""
Sep 18 22:26:51.908755 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.908636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-zpt2l\""
Sep 18 22:26:51.908820 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.908789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 22:26:51.908880 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.908864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 22:26:51.908936 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.908922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 22:26:51.915463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.915440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:26:51.968764 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.968694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5t9c\" (UniqueName: \"kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:51.968917 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:51.968876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.069768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.069740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t9c\" (UniqueName: \"kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.069886 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.069788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.072279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.072258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.082185 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.082162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t9c\" (UniqueName: \"kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c\") pod \"auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.214847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.214791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:52.509577 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.509550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:26:52.510283 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:26:52.510257    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59832bd_a06b_4299_a564_1465393cf1db.slice/crio-fdd3069c61faf00c71bd4c742f7cb1191667313301c4a7a8c3fde6fa5eb531f7 WatchSource:0}: Error finding container fdd3069c61faf00c71bd4c742f7cb1191667313301c4a7a8c3fde6fa5eb531f7: Status 404 returned error can't find the container with id fdd3069c61faf00c71bd4c742f7cb1191667313301c4a7a8c3fde6fa5eb531f7
Sep 18 22:26:52.984122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.984082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" event={"ID":"a59832bd-a06b-4299-a564-1465393cf1db","Type":"ContainerStarted","Data":"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"}
Sep 18 22:26:52.984625 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.984149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" event={"ID":"a59832bd-a06b-4299-a564-1465393cf1db","Type":"ContainerStarted","Data":"fdd3069c61faf00c71bd4c742f7cb1191667313301c4a7a8c3fde6fa5eb531f7"}
Sep 18 22:26:52.984875 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:52.984849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:26:53.015534 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:53.015484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podStartSLOduration=2.015470095 podStartE2EDuration="2.015470095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:53.012966082 +0000 UTC m=+2153.211674380" watchObservedRunningTime="2026-09-18 22:26:53.015470095 +0000 UTC m=+2153.214178474"
Sep 18 22:26:56.877241 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:26:56.877211    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:26:56.877627 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:26:56.877412    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:27:09.876530 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:09.876499    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:27:09.876994 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:27:09.876688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:27:20.877105 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:20.877075    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:27:20.877513 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:27:20.877240    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:27:24.995512 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:24.995485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:27:35.877229 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:35.877200    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:27:35.877632 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:27:35.877363    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:27:48.877203 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:48.877175    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:27:48.877656 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:27:48.877348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:27:59.876927 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:27:59.876896    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:27:59.877308 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:27:59.877089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:28:11.879922 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:11.879892    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:28:11.880283 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:28:11.880055    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:28:24.877027 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:24.876942    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:28:24.877575 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:28:24.877171    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:28:38.877110 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:38.877077    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:28:38.877589 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:28:38.877263    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:28:50.876456 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:50.876429    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:28:50.876828 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:28:50.876610    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:28:53.823697 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:53.823666    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:28:54.995345 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:28:54.995305    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:29:01.878982 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:01.878948    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:01.879480 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:01.879145    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:04.995420 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:04.995363    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:29:08.842699 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:08.842631    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" containerID="cri-o://368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de" gracePeriod=45
Sep 18 22:29:14.877011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:14.876975    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:14.877492 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:14.877188    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:14.995005 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:14.994975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:29:14.995125 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:14.995053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:29:24.995702 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:24.995664    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:29:28.877196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:28.877168    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:28.877589 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:28.877332    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:34.994946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:34.994910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:29:39.086546 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.086522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:29:39.205254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.205183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5t9c\" (UniqueName: \"kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c\") pod \"a59832bd-a06b-4299-a564-1465393cf1db\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") "
Sep 18 22:29:39.205415 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.205267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs\") pod \"a59832bd-a06b-4299-a564-1465393cf1db\" (UID: \"a59832bd-a06b-4299-a564-1465393cf1db\") "
Sep 18 22:29:39.207334 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.207312    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a59832bd-a06b-4299-a564-1465393cf1db" (UID: "a59832bd-a06b-4299-a564-1465393cf1db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:29:39.207446 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.207311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c" (OuterVolumeSpecName: "kube-api-access-q5t9c") pod "a59832bd-a06b-4299-a564-1465393cf1db" (UID: "a59832bd-a06b-4299-a564-1465393cf1db"). InnerVolumeSpecName "kube-api-access-q5t9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:39.306064 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.306038    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a59832bd-a06b-4299-a564-1465393cf1db-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:29:39.306064 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.306059    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5t9c\" (UniqueName: \"kubernetes.io/projected/a59832bd-a06b-4299-a564-1465393cf1db-kube-api-access-q5t9c\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:29:39.720073 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.720046    2563 generic.go:356] "Generic (PLEG): container finished" podID="a59832bd-a06b-4299-a564-1465393cf1db" containerID="368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de" exitCode=0
Sep 18 22:29:39.720208 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.720092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"
Sep 18 22:29:39.720208 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.720135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" event={"ID":"a59832bd-a06b-4299-a564-1465393cf1db","Type":"ContainerDied","Data":"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"}
Sep 18 22:29:39.720208 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.720186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7" event={"ID":"a59832bd-a06b-4299-a564-1465393cf1db","Type":"ContainerDied","Data":"fdd3069c61faf00c71bd4c742f7cb1191667313301c4a7a8c3fde6fa5eb531f7"}
Sep 18 22:29:39.720328 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.720210    2563 scope.go:117] "RemoveContainer" containerID="368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"
Sep 18 22:29:39.729449 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.729431    2563 scope.go:117] "RemoveContainer" containerID="368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"
Sep 18 22:29:39.729713 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:39.729695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de\": container with ID starting with 368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de not found: ID does not exist" containerID="368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"
Sep 18 22:29:39.729762 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.729722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de"} err="failed to get container status \"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de\": rpc error: code = NotFound desc = could not find container \"368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de\": container with ID starting with 368fee38d7a374516699e392e416aa9f6a61a5c47ad5256d4da6b7f1506cb2de not found: ID does not exist"
Sep 18 22:29:39.742552 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.742532    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:29:39.745037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.745015    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6cdd5cf5c6m5bf7"]
Sep 18 22:29:39.876472 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.876449    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:39.876685 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:39.876663    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:39.881436 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:39.881413    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a59832bd-a06b-4299-a564-1465393cf1db" path="/var/lib/kubelet/pods/a59832bd-a06b-4299-a564-1465393cf1db/volumes"
Sep 18 22:29:48.030349 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.030258    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:29:48.030911 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.030744    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main"
Sep 18 22:29:48.030911 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.030763    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main"
Sep 18 22:29:48.030911 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.030864    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a59832bd-a06b-4299-a564-1465393cf1db" containerName="main"
Sep 18 22:29:48.035550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.035527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.039003 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.038981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 22:29:48.039122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.039014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-dm7mz\""
Sep 18 22:29:48.039122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.039040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 22:29:48.039218 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.039119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 22:29:48.039419 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.039385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-h7gmx\""
Sep 18 22:29:48.046473 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.046449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:29:48.168762 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.168725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrkj\" (UniqueName: \"kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.168922 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.168788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.269597 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.269568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrkj\" (UniqueName: \"kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.269732 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.269624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.272040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.272017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.278446 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.278421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrkj\" (UniqueName: \"kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj\") pod \"auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.347936 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.347908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.624528 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.624458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:29:48.627369 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:29:48.627336    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e7e35_51b0_4317_b398_cda8da5d655e.slice/crio-b534d352cf255b35a66e581e9755be416d44bf41ae5df280b4dbf10196ef6c72 WatchSource:0}: Error finding container b534d352cf255b35a66e581e9755be416d44bf41ae5df280b4dbf10196ef6c72: Status 404 returned error can't find the container with id b534d352cf255b35a66e581e9755be416d44bf41ae5df280b4dbf10196ef6c72
Sep 18 22:29:48.763022 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.762993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" event={"ID":"423e7e35-51b0-4317-b398-cda8da5d655e","Type":"ContainerStarted","Data":"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"}
Sep 18 22:29:48.763192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.763042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" event={"ID":"423e7e35-51b0-4317-b398-cda8da5d655e","Type":"ContainerStarted","Data":"b534d352cf255b35a66e581e9755be416d44bf41ae5df280b4dbf10196ef6c72"}
Sep 18 22:29:48.763192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.763096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:29:48.790181 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:48.790130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podStartSLOduration=0.79011458 podStartE2EDuration="790.11458ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:48.789299057 +0000 UTC m=+2328.988007356" watchObservedRunningTime="2026-09-18 22:29:48.79011458 +0000 UTC m=+2328.988822879"
Sep 18 22:29:50.876632 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:50.876602    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:51.779260 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.779234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/9.log"
Sep 18 22:29:51.779676 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.779659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/8.log"
Sep 18 22:29:51.779724 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.779695    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a" exitCode=1
Sep 18 22:29:51.779762 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.779735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"}
Sep 18 22:29:51.779801 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.779782    2563 scope.go:117] "RemoveContainer" containerID="8a12cf3fdc0d68e78242e7d60c1658177f74a1996f202031cde7fbcd8d8dbfe5"
Sep 18 22:29:51.780121 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:51.780096    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:29:51.780318 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:51.780297    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:52.786073 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:52.786046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/9.log"
Sep 18 22:29:54.302065 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:54.302035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:29:54.302513 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:54.302320    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:29:54.302513 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:54.302497    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:29:54.964039 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:54.964010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:29:54.964325 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:29:54.964310    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:29:54.964504 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:29:54.964486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:30:09.881637 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:30:09.881610    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:30:09.882033 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:30:09.881778    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:30:19.771515 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:30:19.771486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:30:21.882619 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:30:21.882588    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:30:21.883374 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:30:21.883341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:30:34.877116 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:30:34.877088    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:30:34.877585 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:30:34.877250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:30:49.876436 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:30:49.876407    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:30:49.876907 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:30:49.876585    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:01.986667 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:01.986638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/9.log"
Sep 18 22:31:01.995302 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:01.995281    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/9.log"
Sep 18 22:31:01.999453 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:01.999431    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:31:02.007477 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:02.007460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:31:04.876501 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:04.876470    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:31:04.877021 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:31:04.876643    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:16.876734 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:16.876657    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:31:16.877103 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:31:16.876862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:31.879630 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:31.879599    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:31:31.880098 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:31:31.879855    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:43.877656 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:43.877622    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:31:43.878123 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:31:43.877790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:52.879311 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:52.879272    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:31:54.876769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:54.876736    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:31:54.877124 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:31:54.876907    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:31:59.771091 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:31:59.771057    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:32:05.876673 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:05.876643    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:32:05.879074 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:05.876819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:32:07.898231 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:07.898172    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main" containerID="cri-o://62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a" gracePeriod=45
Sep 18 22:32:09.771137 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:09.771098    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:32:17.877292 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:17.877258    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:32:17.877918 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:17.877441    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:32:19.771608 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:19.771564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:32:19.772000 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:19.771690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:32:29.771359 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:29.771320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:32:32.876926 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:32.876895    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:32:32.877288 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:32.877059    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:32:38.146790 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.146764    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:32:38.199035 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.198959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs\") pod \"423e7e35-51b0-4317-b398-cda8da5d655e\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") "
Sep 18 22:32:38.199035 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.199007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnrkj\" (UniqueName: \"kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj\") pod \"423e7e35-51b0-4317-b398-cda8da5d655e\" (UID: \"423e7e35-51b0-4317-b398-cda8da5d655e\") "
Sep 18 22:32:38.201058 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.201027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj" (OuterVolumeSpecName: "kube-api-access-dnrkj") pod "423e7e35-51b0-4317-b398-cda8da5d655e" (UID: "423e7e35-51b0-4317-b398-cda8da5d655e"). InnerVolumeSpecName "kube-api-access-dnrkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:32:38.201179 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.201074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "423e7e35-51b0-4317-b398-cda8da5d655e" (UID: "423e7e35-51b0-4317-b398-cda8da5d655e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:32:38.299871 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.299850    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e7e35-51b0-4317-b398-cda8da5d655e-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:32:38.299871 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.299871    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrkj\" (UniqueName: \"kubernetes.io/projected/423e7e35-51b0-4317-b398-cda8da5d655e-kube-api-access-dnrkj\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:32:38.525884 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.525824    2563 generic.go:356] "Generic (PLEG): container finished" podID="423e7e35-51b0-4317-b398-cda8da5d655e" containerID="62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a" exitCode=0
Sep 18 22:32:38.525988 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.525879    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"
Sep 18 22:32:38.525988 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.525914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" event={"ID":"423e7e35-51b0-4317-b398-cda8da5d655e","Type":"ContainerDied","Data":"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"}
Sep 18 22:32:38.526072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.526001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg" event={"ID":"423e7e35-51b0-4317-b398-cda8da5d655e","Type":"ContainerDied","Data":"b534d352cf255b35a66e581e9755be416d44bf41ae5df280b4dbf10196ef6c72"}
Sep 18 22:32:38.526072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.526030    2563 scope.go:117] "RemoveContainer" containerID="62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"
Sep 18 22:32:38.534044 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.534027    2563 scope.go:117] "RemoveContainer" containerID="62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"
Sep 18 22:32:38.534320 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:38.534291    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a\": container with ID starting with 62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a not found: ID does not exist" containerID="62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"
Sep 18 22:32:38.534370 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.534321    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a"} err="failed to get container status \"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a\": rpc error: code = NotFound desc = could not find container \"62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a\": container with ID starting with 62f70be391c0e4a567095ddaefd2f9257b111d24191060f7d749853aaf43b27a not found: ID does not exist"
Sep 18 22:32:38.552227 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.552205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:32:38.556611 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:38.556589    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-549c84f84b-f86hg"]
Sep 18 22:32:39.881609 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:39.881576    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" path="/var/lib/kubelet/pods/423e7e35-51b0-4317-b398-cda8da5d655e/volumes"
Sep 18 22:32:45.877356 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:45.877284    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:32:45.877745 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:45.877496    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:32:58.204148 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.204118    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:32:58.204610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.204420    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main"
Sep 18 22:32:58.204610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.204432    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main"
Sep 18 22:32:58.204610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.204490    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e7e35-51b0-4317-b398-cda8da5d655e" containerName="main"
Sep 18 22:32:58.208886 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.208866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.212319 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.212294    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 22:32:58.212487 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.212470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-td684\""
Sep 18 22:32:58.212550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.212482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 22:32:58.212550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.212493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 18 22:32:58.216822 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.216801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:32:58.357877 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.357877 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.358061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.358061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.358061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.358061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.357993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgpq\" (UniqueName: \"kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459225 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459225 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgpq\" (UniqueName: \"kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459656 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459701 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.459758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.459743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.461541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.461518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.461799 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.461780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.470081 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.470056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgpq\" (UniqueName: \"kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.518867 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.518837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:32:58.812224 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.812145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:32:58.815251 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:32:58.815199    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded404cfa_3522_496c_b930_b5e0d47ca811.slice/crio-a3c61a6e8b9b714097a350c0080bb44267eea47709816d8801f4dd7c1fb7cf17 WatchSource:0}: Error finding container a3c61a6e8b9b714097a350c0080bb44267eea47709816d8801f4dd7c1fb7cf17: Status 404 returned error can't find the container with id a3c61a6e8b9b714097a350c0080bb44267eea47709816d8801f4dd7c1fb7cf17
Sep 18 22:32:58.876436 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:58.876415    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:32:58.876609 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:32:58.876591    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:32:59.621188 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:59.621149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" event={"ID":"ed404cfa-3522-496c-b930-b5e0d47ca811","Type":"ContainerStarted","Data":"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"}
Sep 18 22:32:59.621571 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:59.621212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" event={"ID":"ed404cfa-3522-496c-b930-b5e0d47ca811","Type":"ContainerStarted","Data":"a3c61a6e8b9b714097a350c0080bb44267eea47709816d8801f4dd7c1fb7cf17"}
Sep 18 22:32:59.652738 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:32:59.652672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" podStartSLOduration=1.652651154 podStartE2EDuration="1.652651154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:59.652102079 +0000 UTC m=+2519.850810378" watchObservedRunningTime="2026-09-18 22:32:59.652651154 +0000 UTC m=+2519.851359454"
Sep 18 22:33:00.519609 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:00.519574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:33:08.519412 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:08.519373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:33:08.531281 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:08.531260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:33:08.672086 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:08.672061    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:33:11.878614 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:11.878585    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:33:11.878998 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:33:11.878771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:33:23.876600 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:23.876573    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:33:23.876983 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:33:23.876737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:33:36.877295 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:36.877263    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:33:36.877777 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:33:36.877434    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:33:50.877329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:33:50.877292    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:33:50.877754 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:33:50.877583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:04.876975 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:04.876945    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:04.877350 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:04.877127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:12.819905 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:12.819865    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:34:12.828591 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:12.828556    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:34:12.828591 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:34:12.828591 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:34:12.828591 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c" cmd=["/bin/sleep","15"]
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:12.828605    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" message="" err=<
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:12.828617    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerName="main" containerID="cri-o://fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"
Sep 18 22:34:12.828812 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:12.828661    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerName="main" containerID="cri-o://fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c" gracePeriod=60
Sep 18 22:34:13.178413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.178368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:34:13.194841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.194950 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.194950 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.194950 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.194950 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xgpq\" (UniqueName: \"kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.195152 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.194988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm\") pod \"ed404cfa-3522-496c-b930-b5e0d47ca811\" (UID: \"ed404cfa-3522-496c-b930-b5e0d47ca811\") "
Sep 18 22:34:13.195152 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.195124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:13.195152 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.195144    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:13.195308 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.195262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home" (OuterVolumeSpecName: "home") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:13.197133 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.197110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm" (OuterVolumeSpecName: "dshm") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:13.197215 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.197136    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:34:13.197215 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.197184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq" (OuterVolumeSpecName: "kube-api-access-8xgpq") pod "ed404cfa-3522-496c-b930-b5e0d47ca811" (UID: "ed404cfa-3522-496c-b930-b5e0d47ca811"). InnerVolumeSpecName "kube-api-access-8xgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:13.296448 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296419    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.296448 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296444    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed404cfa-3522-496c-b930-b5e0d47ca811-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.296610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgpq\" (UniqueName: \"kubernetes.io/projected/ed404cfa-3522-496c-b930-b5e0d47ca811-kube-api-access-8xgpq\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.296610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296462    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.296610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296471    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.296610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.296478    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed404cfa-3522-496c-b930-b5e0d47ca811-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:34:13.949731 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.949697    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerID="fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c" exitCode=0
Sep 18 22:34:13.950193 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.949754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" event={"ID":"ed404cfa-3522-496c-b930-b5e0d47ca811","Type":"ContainerDied","Data":"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"}
Sep 18 22:34:13.950324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.950221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6" event={"ID":"ed404cfa-3522-496c-b930-b5e0d47ca811","Type":"ContainerDied","Data":"a3c61a6e8b9b714097a350c0080bb44267eea47709816d8801f4dd7c1fb7cf17"}
Sep 18 22:34:13.950324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.949766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"
Sep 18 22:34:13.950324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.950257    2563 scope.go:117] "RemoveContainer" containerID="fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"
Sep 18 22:34:13.959290 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.959275    2563 scope.go:117] "RemoveContainer" containerID="fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"
Sep 18 22:34:13.959974 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:13.959950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c\": container with ID starting with fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c not found: ID does not exist" containerID="fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"
Sep 18 22:34:13.960149 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.960129    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c"} err="failed to get container status \"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c\": rpc error: code = NotFound desc = could not find container \"fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c\": container with ID starting with fd18d2d68a0394238b20eba1cb056f65860d8e501cd27e4eb024bd3066acd20c not found: ID does not exist"
Sep 18 22:34:13.991508 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.991487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:34:13.994448 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:13.994429    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mkbp6"]
Sep 18 22:34:15.882530 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:15.882450    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" path="/var/lib/kubelet/pods/ed404cfa-3522-496c-b930-b5e0d47ca811/volumes"
Sep 18 22:34:19.877093 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:19.877060    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:19.877498 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:19.877300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:30.389041 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.389005    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:34:30.389439 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.389305    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerName="main"
Sep 18 22:34:30.389439 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.389317    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerName="main"
Sep 18 22:34:30.389439 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.389382    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed404cfa-3522-496c-b930-b5e0d47ca811" containerName="main"
Sep 18 22:34:30.393565 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.393544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.397246 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.397221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-vssc6\""
Sep 18 22:34:30.397246 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.397238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 22:34:30.397464 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.397226    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 22:34:30.397464 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.397226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 22:34:30.397464 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.397230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-dpvjp\""
Sep 18 22:34:30.401132 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.401008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:34:30.525877 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.525845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmfl\" (UniqueName: \"kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.526029 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.525889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.627259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.627230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmfl\" (UniqueName: \"kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.627384 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.627276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.629770 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.629750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.635729 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.635704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmfl\" (UniqueName: \"kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.705645 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.705579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:30.984545 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:30.984514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:34:30.986369 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:34:30.986334    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0297bf2a_1968_484f_b900_4e95455c0495.slice/crio-fdf0165c994ddbd43fb37d63a651452256a09e0f4e0033faff0de114b97e86cf WatchSource:0}: Error finding container fdf0165c994ddbd43fb37d63a651452256a09e0f4e0033faff0de114b97e86cf: Status 404 returned error can't find the container with id fdf0165c994ddbd43fb37d63a651452256a09e0f4e0033faff0de114b97e86cf
Sep 18 22:34:31.025590 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:31.025414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" event={"ID":"0297bf2a-1968-484f-b900-4e95455c0495","Type":"ContainerStarted","Data":"fdf0165c994ddbd43fb37d63a651452256a09e0f4e0033faff0de114b97e86cf"}
Sep 18 22:34:32.033132 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:32.033100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" event={"ID":"0297bf2a-1968-484f-b900-4e95455c0495","Type":"ContainerStarted","Data":"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"}
Sep 18 22:34:32.033606 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:32.033226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:34:32.060271 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:32.060218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podStartSLOduration=2.060204469 podStartE2EDuration="2.060204469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:32.057978709 +0000 UTC m=+2612.256687009" watchObservedRunningTime="2026-09-18 22:34:32.060204469 +0000 UTC m=+2612.258912767"
Sep 18 22:34:32.876879 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:32.876851    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:32.877039 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:32.877020    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:43.883083 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:43.883058    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:43.883474 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:43.883219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:56.877102 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:56.877070    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:57.144533 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.144465    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/10.log"
Sep 18 22:34:57.144905 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.144889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/9.log"
Sep 18 22:34:57.144964 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.144924    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0" exitCode=1
Sep 18 22:34:57.145033 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.145008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"}
Sep 18 22:34:57.145100 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.145086    2563 scope.go:117] "RemoveContainer" containerID="3df397e0a55cd7c394bfcf07f49c56e7e5d4370dadd856cd280cccaba327b95a"
Sep 18 22:34:57.145440 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:57.145419    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:34:57.145628 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:34:57.145608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:34:58.150972 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:34:58.150945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/10.log"
Sep 18 22:35:03.040016 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:03.039981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:35:04.301960 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:04.301926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:35:04.302339 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:04.302266    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:04.302468 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:04.302448    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:35:04.964081 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:04.964035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:35:05.181543 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:05.181516    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:05.181710 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:05.181683    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:35:19.876954 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:19.876922    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:19.877439 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:19.877129    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:35:32.876972 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:32.876942    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:32.877364 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:32.877124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:35:43.876642 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:43.876606    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:43.877107 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:43.876836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:35:55.877044 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:35:55.876959    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:35:55.877526 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:35:55.877186    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:36:02.013204 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:02.013179    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/10.log"
Sep 18 22:36:02.022815 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:02.022792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/10.log"
Sep 18 22:36:02.026056 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:02.026037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:36:02.035584 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:02.035568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:36:10.876680 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:10.876651    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:36:10.877062 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:36:10.876820    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:36:23.877329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:23.877284    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:36:23.877834 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:36:23.877574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:36:35.877488 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:35.877454    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:36:35.877948 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:36:35.877681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:36:36.027223 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:36.027192    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:36:43.040006 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:43.039962    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:36:50.877527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:50.877488    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:36:50.877970 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:36:50.877671    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:36:51.046763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:51.046726    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main" containerID="cri-o://ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae" gracePeriod=45
Sep 18 22:36:53.039312 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:36:53.039272    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:37:01.877926 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:01.877899    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:37:01.878343 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:01.878089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:37:03.039547 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:03.039513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:37:03.039918 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:03.039604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:37:13.040026 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:13.039985    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:37:15.876872 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:15.876794    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:37:15.877297 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:15.877001    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:37:21.287902 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.287875    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:37:21.428042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.428015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmfl\" (UniqueName: \"kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl\") pod \"0297bf2a-1968-484f-b900-4e95455c0495\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") "
Sep 18 22:37:21.428175 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.428074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs\") pod \"0297bf2a-1968-484f-b900-4e95455c0495\" (UID: \"0297bf2a-1968-484f-b900-4e95455c0495\") "
Sep 18 22:37:21.430130 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.430099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0297bf2a-1968-484f-b900-4e95455c0495" (UID: "0297bf2a-1968-484f-b900-4e95455c0495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:37:21.430231 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.430173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl" (OuterVolumeSpecName: "kube-api-access-kbmfl") pod "0297bf2a-1968-484f-b900-4e95455c0495" (UID: "0297bf2a-1968-484f-b900-4e95455c0495"). InnerVolumeSpecName "kube-api-access-kbmfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:37:21.528560 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.528535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmfl\" (UniqueName: \"kubernetes.io/projected/0297bf2a-1968-484f-b900-4e95455c0495-kube-api-access-kbmfl\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:37:21.528560 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.528556    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0297bf2a-1968-484f-b900-4e95455c0495-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:37:21.790891 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.790811    2563 generic.go:356] "Generic (PLEG): container finished" podID="0297bf2a-1968-484f-b900-4e95455c0495" containerID="ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae" exitCode=0
Sep 18 22:37:21.790891 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.790852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"
Sep 18 22:37:21.791076 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.790895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" event={"ID":"0297bf2a-1968-484f-b900-4e95455c0495","Type":"ContainerDied","Data":"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"}
Sep 18 22:37:21.791076 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.790943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8" event={"ID":"0297bf2a-1968-484f-b900-4e95455c0495","Type":"ContainerDied","Data":"fdf0165c994ddbd43fb37d63a651452256a09e0f4e0033faff0de114b97e86cf"}
Sep 18 22:37:21.791076 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.790967    2563 scope.go:117] "RemoveContainer" containerID="ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"
Sep 18 22:37:21.799772 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.799752    2563 scope.go:117] "RemoveContainer" containerID="ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"
Sep 18 22:37:21.800037 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:21.800015    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae\": container with ID starting with ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae not found: ID does not exist" containerID="ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"
Sep 18 22:37:21.800103 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.800046    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae"} err="failed to get container status \"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae\": rpc error: code = NotFound desc = could not find container \"ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae\": container with ID starting with ee33841886f0ac7e2424a50ccaec88437b2f42017f901d910f32af1f8a1f32ae not found: ID does not exist"
Sep 18 22:37:21.820294 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.820268    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:37:21.826608 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.826585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schebc6b8"]
Sep 18 22:37:21.881346 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:21.881322    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0297bf2a-1968-484f-b900-4e95455c0495" path="/var/lib/kubelet/pods/0297bf2a-1968-484f-b900-4e95455c0495/volumes"
Sep 18 22:37:29.877303 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:29.877275    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:37:29.877795 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:29.877450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:37:34.133952 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.133921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:37:34.134442 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.134358    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main"
Sep 18 22:37:34.134442 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.134376    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main"
Sep 18 22:37:34.134583 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.134498    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0297bf2a-1968-484f-b900-4e95455c0495" containerName="main"
Sep 18 22:37:34.141666 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.141632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.144068 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.144035    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 22:37:34.144207 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.144080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-wk7pc\""
Sep 18 22:37:34.144207 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.144102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 22:37:34.144207 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.144121    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 22:37:34.156684 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.156660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:37:34.321695 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.321695 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmqc\" (UniqueName: \"kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.321888 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.321888 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.321888 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.321888 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.322056 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.321930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.422796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.422796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.422796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmqc\" (UniqueName: \"kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.422916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423286 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.423260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423286 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.423280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423422 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.423353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.423486 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.423419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.425128 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.425102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.425231 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.425178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.432535 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.432514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmqc\" (UniqueName: \"kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.452479 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.452461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:37:34.559105 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.559077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:37:34.562692 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.562666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.564946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.564915    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-mdscj\""
Sep 18 22:37:34.574826 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.574803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:37:34.725715 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.725638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.725715 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.725701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjfw\" (UniqueName: \"kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.745122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.745092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:37:34.791009 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.790989    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:37:34.826444 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.826418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.826533 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.826473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjfw\" (UniqueName: \"kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.828612 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.828589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.845188 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.845171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjfw\" (UniqueName: \"kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:34.851710 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.851680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerStarted","Data":"e0cdaaa0a799ee7439c61b761f77785bfbe51cd0cefab1eb59fc42314043f4a8"}
Sep 18 22:37:34.874932 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:34.874914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:35.150660 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:35.150631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:37:35.153947 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:37:35.153919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc65fac_11cf_4831_b36b_f8321f2e0b93.slice/crio-e86fe608e7346aae5fb25766c68bacdf5ac22d829d64b1ffe391d192e3279432 WatchSource:0}: Error finding container e86fe608e7346aae5fb25766c68bacdf5ac22d829d64b1ffe391d192e3279432: Status 404 returned error can't find the container with id e86fe608e7346aae5fb25766c68bacdf5ac22d829d64b1ffe391d192e3279432
Sep 18 22:37:35.862811 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:35.862765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" event={"ID":"ebc65fac-11cf-4831-b36b-f8321f2e0b93","Type":"ContainerStarted","Data":"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"}
Sep 18 22:37:35.863009 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:35.862827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" event={"ID":"ebc65fac-11cf-4831-b36b-f8321f2e0b93","Type":"ContainerStarted","Data":"e86fe608e7346aae5fb25766c68bacdf5ac22d829d64b1ffe391d192e3279432"}
Sep 18 22:37:35.863083 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:35.863043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:37:35.894923 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:35.894863    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podStartSLOduration=1.894841316 podStartE2EDuration="1.894841316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:37:35.892942579 +0000 UTC m=+2796.091650880" watchObservedRunningTime="2026-09-18 22:37:35.894841316 +0000 UTC m=+2796.093549616"
Sep 18 22:37:41.878250 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:41.878226    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:37:41.878622 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:41.878410    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:37:41.895025 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:41.895001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerStarted","Data":"8111338668f2e50ddac4db25ef411c6a64c9ab9ef62da24a11e637ab74ddcd52"}
Sep 18 22:37:42.900971 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:42.900935    2563 generic.go:356] "Generic (PLEG): container finished" podID="790a21af-0756-478e-95cc-9c81d821f8d3" containerID="8111338668f2e50ddac4db25ef411c6a64c9ab9ef62da24a11e637ab74ddcd52" exitCode=0
Sep 18 22:37:42.901448 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:42.901007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerDied","Data":"8111338668f2e50ddac4db25ef411c6a64c9ab9ef62da24a11e637ab74ddcd52"}
Sep 18 22:37:54.876809 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:37:54.876768    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:37:54.877420 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:37:54.877022    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:38:06.870347 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:06.870308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:38:08.877351 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:08.877319    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:38:08.877892 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:38:08.877583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:38:12.066877 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:12.066841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerStarted","Data":"8640ad6f91d1993a72973aeb92aeca46a8e8cf5c0d58f2270b862c2cc620e5ea"}
Sep 18 22:38:13.074029 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:13.073995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerStarted","Data":"b1789862270981c786c82fb901ad63f298c12e0cb2f1bfa80c97413d254b01d2"}
Sep 18 22:38:13.099435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:13.099370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podStartSLOduration=3.144633103 podStartE2EDuration="39.099354406s" totalImagesPullingTime="34.868681759s" totalInitContainerRuntime="1.086039544s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:37:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:38:13.097050459 +0000 UTC m=+2833.295758761" watchObservedRunningTime="2026-09-18 22:38:13.099354406 +0000 UTC m=+2833.298062705"
Sep 18 22:38:13.453067 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:13.453033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:38:14.453644 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:14.453611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:38:14.455151 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:14.455122    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:38:21.880207 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:21.880172    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:38:21.880660 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:38:21.880386    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:38:24.452984 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:24.452938    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:38:34.453107 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:34.453057    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:38:34.876584 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:34.876551    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:38:34.876875 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:38:34.876764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:38:44.453269 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:44.453229    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:38:48.877308 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:48.877275    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:38:48.877713 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:38:48.877495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:38:54.453846 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:38:54.453804    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:02.876315 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:02.876281    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:39:02.876856 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:39:02.876524    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:39:04.453813 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:04.453773    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:14.453482 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:14.453440    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:17.877093 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:17.877063    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:39:17.877559 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:39:17.877238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:39:24.453433 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:24.453374    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:28.877216 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:28.877182    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:39:28.877616 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:39:28.877368    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:39:34.453715 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:34.453674    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:39.877247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:39.877212    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:39:39.877770 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:39:39.877492    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:39:44.452883 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:44.452841    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:39:51.878312 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:51.878284    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:39:51.878776 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:39:51.878485    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:39:54.453660 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:39:54.453613    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:40:02.877026 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:02.876995    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:40:03.605664 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.605636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:40:03.606041 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.606026    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/10.log"
Sep 18 22:40:03.606096 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.606061    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832" exitCode=1
Sep 18 22:40:03.606142 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.606104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"}
Sep 18 22:40:03.606264 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.606155    2563 scope.go:117] "RemoveContainer" containerID="a51dab8c839fbdae559e2dc52562990e54b7c60821483bfb09baf321ad67acf0"
Sep 18 22:40:03.606504 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:03.606482    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:03.606719 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:03.606697    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:04.301615 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.301583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:40:04.469214 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.469181    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:40:04.481984 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.481958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:40:04.613429 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.613406    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:40:04.613746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.613732    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:04.613914 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:04.613896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:04.964571 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:04.964484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:40:05.618252 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:05.618223    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:05.618652 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:05.618386    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:12.073194 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.073159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:40:12.073697 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.073508    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar" containerID="cri-o://b1789862270981c786c82fb901ad63f298c12e0cb2f1bfa80c97413d254b01d2" gracePeriod=60
Sep 18 22:40:12.080302 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.080260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:40:12.654663 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.654636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx_790a21af-0756-478e-95cc-9c81d821f8d3/modelcar/0.log"
Sep 18 22:40:12.655315 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.655292    2563 generic.go:356] "Generic (PLEG): container finished" podID="790a21af-0756-478e-95cc-9c81d821f8d3" containerID="b1789862270981c786c82fb901ad63f298c12e0cb2f1bfa80c97413d254b01d2" exitCode=143
Sep 18 22:40:12.655388 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:12.655365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerDied","Data":"b1789862270981c786c82fb901ad63f298c12e0cb2f1bfa80c97413d254b01d2"}
Sep 18 22:40:16.869368 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:16.869279    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:18.877136 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:18.877098    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:18.877626 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:18.877310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:26.869648 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:26.869612    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:27.096634 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.096582    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" containerID="cri-o://8640ad6f91d1993a72973aeb92aeca46a8e8cf5c0d58f2270b862c2cc620e5ea" gracePeriod=45
Sep 18 22:40:27.098946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.098918    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" containerID="cri-o://80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2" gracePeriod=45
Sep 18 22:40:27.470067 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.470025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 18 22:40:27.734415 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.734334    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx_790a21af-0756-478e-95cc-9c81d821f8d3/modelcar/0.log"
Sep 18 22:40:27.735130 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.735104    2563 generic.go:356] "Generic (PLEG): container finished" podID="790a21af-0756-478e-95cc-9c81d821f8d3" containerID="8640ad6f91d1993a72973aeb92aeca46a8e8cf5c0d58f2270b862c2cc620e5ea" exitCode=0
Sep 18 22:40:27.735243 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.735159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerDied","Data":"8640ad6f91d1993a72973aeb92aeca46a8e8cf5c0d58f2270b862c2cc620e5ea"}
Sep 18 22:40:27.866479 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.866461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx_790a21af-0756-478e-95cc-9c81d821f8d3/modelcar/0.log"
Sep 18 22:40:27.869326 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.869306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:40:27.997485 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997485 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbmqc\" (UniqueName: \"kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir\") pod \"790a21af-0756-478e-95cc-9c81d821f8d3\" (UID: \"790a21af-0756-478e-95cc-9c81d821f8d3\") "
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:27.997892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:27.998263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997954    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-kserve-provision-location\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:27.998263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.997985    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:27.998378 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.998349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home" (OuterVolumeSpecName: "home") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:27.999849 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.999822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:27.999964 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:27.999920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm" (OuterVolumeSpecName: "dshm") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:28.000167 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.000148    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc" (OuterVolumeSpecName: "kube-api-access-rbmqc") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "kube-api-access-rbmqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:40:28.009348 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.009324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "790a21af-0756-478e-95cc-9c81d821f8d3" (UID: "790a21af-0756-478e-95cc-9c81d821f8d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:28.099321 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.099291    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:28.099321 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.099315    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmqc\" (UniqueName: \"kubernetes.io/projected/790a21af-0756-478e-95cc-9c81d821f8d3-kube-api-access-rbmqc\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:28.099517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.099326    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790a21af-0756-478e-95cc-9c81d821f8d3-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:28.099517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.099336    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:28.099517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.099343    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790a21af-0756-478e-95cc-9c81d821f8d3-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:28.741236 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.741210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx_790a21af-0756-478e-95cc-9c81d821f8d3/modelcar/0.log"
Sep 18 22:40:28.741986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.741960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx" event={"ID":"790a21af-0756-478e-95cc-9c81d821f8d3","Type":"ContainerDied","Data":"e0cdaaa0a799ee7439c61b761f77785bfbe51cd0cefab1eb59fc42314043f4a8"}
Sep 18 22:40:28.742118 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.742015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"
Sep 18 22:40:28.742118 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.742036    2563 scope.go:117] "RemoveContainer" containerID="b1789862270981c786c82fb901ad63f298c12e0cb2f1bfa80c97413d254b01d2"
Sep 18 22:40:28.751001 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.750983    2563 scope.go:117] "RemoveContainer" containerID="8640ad6f91d1993a72973aeb92aeca46a8e8cf5c0d58f2270b862c2cc620e5ea"
Sep 18 22:40:28.758279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.758258    2563 scope.go:117] "RemoveContainer" containerID="8111338668f2e50ddac4db25ef411c6a64c9ab9ef62da24a11e637ab74ddcd52"
Sep 18 22:40:28.773292 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.773270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:40:28.780773 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:28.780752    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-67677fd99-xkjxx"]
Sep 18 22:40:29.882253 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:29.882212    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" path="/var/lib/kubelet/pods/790a21af-0756-478e-95cc-9c81d821f8d3/volumes"
Sep 18 22:40:33.876646 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:33.876617    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:33.877094 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:33.876770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:36.869896 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:36.869859    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:36.870302 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:36.869965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:40:44.877312 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:44.877282    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:44.877802 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:44.877485    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:46.870126 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:46.870089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:55.877313 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:55.877280    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:40:55.877818 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:55.877503    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:40:56.869536 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:56.869498    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:40:57.335889 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.335865    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:40:57.406596 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.406570    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbjfw\" (UniqueName: \"kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw\") pod \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") "
Sep 18 22:40:57.406725 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.406641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs\") pod \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\" (UID: \"ebc65fac-11cf-4831-b36b-f8321f2e0b93\") "
Sep 18 22:40:57.408571 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.408541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebc65fac-11cf-4831-b36b-f8321f2e0b93" (UID: "ebc65fac-11cf-4831-b36b-f8321f2e0b93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:57.408670 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.408628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw" (OuterVolumeSpecName: "kube-api-access-fbjfw") pod "ebc65fac-11cf-4831-b36b-f8321f2e0b93" (UID: "ebc65fac-11cf-4831-b36b-f8321f2e0b93"). InnerVolumeSpecName "kube-api-access-fbjfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:40:57.508011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.507957    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc65fac-11cf-4831-b36b-f8321f2e0b93-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:57.508011 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.507978    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjfw\" (UniqueName: \"kubernetes.io/projected/ebc65fac-11cf-4831-b36b-f8321f2e0b93-kube-api-access-fbjfw\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:40:57.878644 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.878613    2563 generic.go:356] "Generic (PLEG): container finished" podID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerID="80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2" exitCode=0
Sep 18 22:40:57.878831 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.878712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"
Sep 18 22:40:57.882196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.882172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" event={"ID":"ebc65fac-11cf-4831-b36b-f8321f2e0b93","Type":"ContainerDied","Data":"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"}
Sep 18 22:40:57.882350 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.882207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn" event={"ID":"ebc65fac-11cf-4831-b36b-f8321f2e0b93","Type":"ContainerDied","Data":"e86fe608e7346aae5fb25766c68bacdf5ac22d829d64b1ffe391d192e3279432"}
Sep 18 22:40:57.882350 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.882236    2563 scope.go:117] "RemoveContainer" containerID="80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"
Sep 18 22:40:57.891025 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.891007    2563 scope.go:117] "RemoveContainer" containerID="80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"
Sep 18 22:40:57.891295 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:40:57.891278    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2\": container with ID starting with 80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2 not found: ID does not exist" containerID="80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"
Sep 18 22:40:57.891345 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.891303    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2"} err="failed to get container status \"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2\": rpc error: code = NotFound desc = could not find container \"80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2\": container with ID starting with 80d3d3d87dc90813f06e9de0b0d3534cbb16bb2e5255a55aad5359cd4938eac2 not found: ID does not exist"
Sep 18 22:40:57.905608 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.905584    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:40:57.908969 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:57.908950    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schebrmwn"]
Sep 18 22:40:59.881930 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:40:59.881902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" path="/var/lib/kubelet/pods/ebc65fac-11cf-4831-b36b-f8321f2e0b93/volumes"
Sep 18 22:41:00.610003 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.609974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:41:00.610283 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610271    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610285    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610300    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610313    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar-init"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610318    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar-init"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610324    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar"
Sep 18 22:41:00.610329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar"
Sep 18 22:41:00.610595 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610377    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="modelcar"
Sep 18 22:41:00.610595 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610385    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc65fac-11cf-4831-b36b-f8321f2e0b93" containerName="main"
Sep 18 22:41:00.610595 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.610407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="790a21af-0756-478e-95cc-9c81d821f8d3" containerName="main"
Sep 18 22:41:00.614537 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.614519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.617978 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.617955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 22:41:00.618110 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.617987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 22:41:00.618110 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.618003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tnktd\""
Sep 18 22:41:00.618226 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.618165    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 22:41:00.618267 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.618164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-v8grv\""
Sep 18 22:41:00.629523 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.629495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.629641 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.629544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndcz\" (UniqueName: \"kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.638343 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.638316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:41:00.730905 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.730881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.731036 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.730918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndcz\" (UniqueName: \"kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.733354 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.733332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.741446 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.741425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndcz\" (UniqueName: \"kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-86856gl57\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:00.924940 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:00.924868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:01.209814 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:01.209784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:41:01.211563 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:41:01.211529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b2881_beee_48fa_83da_8b9b8dcdebcb.slice/crio-eddd9695e4d1ec34224300f5f50cbdde096a8834c32c0f7e60fc5e7833b5fdeb WatchSource:0}: Error finding container eddd9695e4d1ec34224300f5f50cbdde096a8834c32c0f7e60fc5e7833b5fdeb: Status 404 returned error can't find the container with id eddd9695e4d1ec34224300f5f50cbdde096a8834c32c0f7e60fc5e7833b5fdeb
Sep 18 22:41:01.898709 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:01.898675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" event={"ID":"413b2881-beee-48fa-83da-8b9b8dcdebcb","Type":"ContainerStarted","Data":"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"}
Sep 18 22:41:01.898866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:01.898731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" event={"ID":"413b2881-beee-48fa-83da-8b9b8dcdebcb","Type":"ContainerStarted","Data":"eddd9695e4d1ec34224300f5f50cbdde096a8834c32c0f7e60fc5e7833b5fdeb"}
Sep 18 22:41:01.898866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:01.898762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:01.925057 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:01.925012    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podStartSLOduration=1.9249970109999999 podStartE2EDuration="1.924997011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:41:01.922009061 +0000 UTC m=+3002.120717360" watchObservedRunningTime="2026-09-18 22:41:01.924997011 +0000 UTC m=+3002.123705310"
Sep 18 22:41:02.042352 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:02.042317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:41:02.055324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:02.055282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:41:02.061596 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:02.061572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:41:02.070884 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:02.070864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:41:08.877203 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:08.877175    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:41:08.877546 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:41:08.877326    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:41:22.877061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:22.877028    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:41:22.877560 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:41:22.877196    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:41:32.904825 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:32.904793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:41:33.877179 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:33.877148    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:41:33.877379 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:41:33.877355    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:41:48.876760 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:48.876676    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:41:48.877180 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:41:48.876877    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:41:58.757412 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.757358    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:41:58.762415 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.761978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.765986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.765960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-pdz4k\""
Sep 18 22:41:58.766139 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.766094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 22:41:58.766200 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.766184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-6ltmm\""
Sep 18 22:41:58.766256 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.766206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 22:41:58.766256 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.766099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 22:41:58.773243 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.773217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:41:58.864710 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.864680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6rz\" (UniqueName: \"kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.864853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.864767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.965465 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.965431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6rz\" (UniqueName: \"kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.965623 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.965484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.967990 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.967962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:58.973119 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:58.973100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6rz\" (UniqueName: \"kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz\") pod \"custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:59.076405 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:59.076325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:41:59.362341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:41:59.362307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:41:59.365548 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:41:59.365523    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbc5d5e_2f1d_4d6a_9b15_663caa45c63e.slice/crio-cd52c39f3552a587c3f7dcc6689091f6d6cfb29d8a2983f207bd70f8756ac179 WatchSource:0}: Error finding container cd52c39f3552a587c3f7dcc6689091f6d6cfb29d8a2983f207bd70f8756ac179: Status 404 returned error can't find the container with id cd52c39f3552a587c3f7dcc6689091f6d6cfb29d8a2983f207bd70f8756ac179
Sep 18 22:42:00.151942 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:00.151902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" event={"ID":"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e","Type":"ContainerStarted","Data":"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"}
Sep 18 22:42:00.152366 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:00.151967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" event={"ID":"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e","Type":"ContainerStarted","Data":"cd52c39f3552a587c3f7dcc6689091f6d6cfb29d8a2983f207bd70f8756ac179"}
Sep 18 22:42:00.152366 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:00.152001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:42:00.177338 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:00.177290    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podStartSLOduration=2.177274992 podStartE2EDuration="2.177274992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:00.176068054 +0000 UTC m=+3060.374776355" watchObservedRunningTime="2026-09-18 22:42:00.177274992 +0000 UTC m=+3060.375983291"
Sep 18 22:42:01.413897 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:01.413859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:42:02.904690 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:02.904660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:42:03.876818 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:03.876790    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:42:03.877050 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:03.876946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:42:12.904630 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:12.904590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:42:14.876819 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:14.876788    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:42:14.879076 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:14.876952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:42:16.433191 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:16.433149    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" containerID="cri-o://acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3" gracePeriod=45
Sep 18 22:42:22.903883 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:22.903847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:42:22.904263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:22.903955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:42:25.877454 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:25.877389    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:42:25.877838 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:25.877593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:42:31.158813 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:31.158782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:42:32.904205 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:32.904167    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:42:37.877368 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:37.877336    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:42:37.877840 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:37.877525    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:42:42.904821 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:42.904781    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:42:46.672344 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.672318    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:42:46.718854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.718795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs\") pod \"413b2881-beee-48fa-83da-8b9b8dcdebcb\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") "
Sep 18 22:42:46.718854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.718836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cndcz\" (UniqueName: \"kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz\") pod \"413b2881-beee-48fa-83da-8b9b8dcdebcb\" (UID: \"413b2881-beee-48fa-83da-8b9b8dcdebcb\") "
Sep 18 22:42:46.720965 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.720942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "413b2881-beee-48fa-83da-8b9b8dcdebcb" (UID: "413b2881-beee-48fa-83da-8b9b8dcdebcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:46.721074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.720965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz" (OuterVolumeSpecName: "kube-api-access-cndcz") pod "413b2881-beee-48fa-83da-8b9b8dcdebcb" (UID: "413b2881-beee-48fa-83da-8b9b8dcdebcb"). InnerVolumeSpecName "kube-api-access-cndcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:46.819385 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.819361    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/413b2881-beee-48fa-83da-8b9b8dcdebcb-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:42:46.819385 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:46.819383    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cndcz\" (UniqueName: \"kubernetes.io/projected/413b2881-beee-48fa-83da-8b9b8dcdebcb-kube-api-access-cndcz\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:42:47.359131 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.359098    2563 generic.go:356] "Generic (PLEG): container finished" podID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerID="acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3" exitCode=0
Sep 18 22:42:47.359287 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.359147    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"
Sep 18 22:42:47.359287 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.359190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" event={"ID":"413b2881-beee-48fa-83da-8b9b8dcdebcb","Type":"ContainerDied","Data":"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"}
Sep 18 22:42:47.359287 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.359246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57" event={"ID":"413b2881-beee-48fa-83da-8b9b8dcdebcb","Type":"ContainerDied","Data":"eddd9695e4d1ec34224300f5f50cbdde096a8834c32c0f7e60fc5e7833b5fdeb"}
Sep 18 22:42:47.359287 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.359269    2563 scope.go:117] "RemoveContainer" containerID="acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"
Sep 18 22:42:47.368658 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.368642    2563 scope.go:117] "RemoveContainer" containerID="acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"
Sep 18 22:42:47.368915 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:47.368898    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3\": container with ID starting with acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3 not found: ID does not exist" containerID="acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"
Sep 18 22:42:47.368962 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.368923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3"} err="failed to get container status \"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3\": rpc error: code = NotFound desc = could not find container \"acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3\": container with ID starting with acb25b5eec440b2f6aeb6663a2b7611676baa9bf72795443b7a1851399dd97e3 not found: ID does not exist"
Sep 18 22:42:47.380984 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.380953    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:42:47.386612 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.386590    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-86856gl57"]
Sep 18 22:42:47.882179 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:47.882145    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" path="/var/lib/kubelet/pods/413b2881-beee-48fa-83da-8b9b8dcdebcb/volumes"
Sep 18 22:42:51.882568 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:51.882540    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:42:51.882964 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:42:51.882748    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:42:58.360037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.360001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:42:58.360560 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.360483    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main"
Sep 18 22:42:58.360560 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.360509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main"
Sep 18 22:42:58.360697 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.360589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="413b2881-beee-48fa-83da-8b9b8dcdebcb" containerName="main"
Sep 18 22:42:58.364939 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.364920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.368060 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.368041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 22:42:58.368185 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.368162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 22:42:58.368255 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.368044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 22:42:58.368255 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.368044    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-87rmk\""
Sep 18 22:42:58.368372 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.368122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7wfdb\""
Sep 18 22:42:58.375824 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.375805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:42:58.411517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.411489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.411624 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.411578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2vp\" (UniqueName: \"kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.512225 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.512197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.512329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.512243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2vp\" (UniqueName: \"kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.514632 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.514613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.520066 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.520030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2vp\" (UniqueName: \"kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp\") pod \"tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.676308 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.676240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:58.957809 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:58.957722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:42:58.961197 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:42:58.961166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda821776_961c_4f2f_8be2_b817450eeb20.slice/crio-a9b1ce319b436a6ff24e756452b0447d2d4e16b59ad6a88c7268b2120ec17aa9 WatchSource:0}: Error finding container a9b1ce319b436a6ff24e756452b0447d2d4e16b59ad6a88c7268b2120ec17aa9: Status 404 returned error can't find the container with id a9b1ce319b436a6ff24e756452b0447d2d4e16b59ad6a88c7268b2120ec17aa9
Sep 18 22:42:59.419252 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:59.419214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" event={"ID":"da821776-961c-4f2f-8be2-b817450eeb20","Type":"ContainerStarted","Data":"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"}
Sep 18 22:42:59.419735 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:59.419276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" event={"ID":"da821776-961c-4f2f-8be2-b817450eeb20","Type":"ContainerStarted","Data":"a9b1ce319b436a6ff24e756452b0447d2d4e16b59ad6a88c7268b2120ec17aa9"}
Sep 18 22:42:59.419735 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:59.419337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:42:59.438077 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:42:59.438031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podStartSLOduration=1.438014727 podStartE2EDuration="1.438014727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:59.436390882 +0000 UTC m=+3119.635099181" watchObservedRunningTime="2026-09-18 22:42:59.438014727 +0000 UTC m=+3119.636723025"
Sep 18 22:43:04.876768 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:04.876735    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:43:04.877163 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:43:04.876914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:43:18.876358 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:18.876290    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:43:18.876815 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:43:18.876477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:43:29.877051 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:29.877018    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:43:29.877521 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:43:29.877169    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:43:30.425732 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:30.425702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:43:36.072960 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:36.072928    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:43:40.425555 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:40.425513    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:43:42.877088 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:42.877059    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:43:42.877477 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:43:42.877230    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:43:50.425986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:50.425949    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:43:51.092209 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:51.092173    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" containerID="cri-o://1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813" gracePeriod=45
Sep 18 22:43:53.876986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:43:53.876949    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:43:53.877465 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:43:53.877123    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:44:00.425850 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:00.425814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:00.426234 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:00.425917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:44:04.877141 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:04.877106    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:44:04.877585 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:04.877289    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:44:10.425459 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:10.425419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:15.876752 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:15.876723    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:44:15.877158 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:15.876911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:44:20.425802 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:20.425766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:21.344690 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.344666    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:44:21.409788 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.409721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs\") pod \"da821776-961c-4f2f-8be2-b817450eeb20\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") "
Sep 18 22:44:21.409910 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.409819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf2vp\" (UniqueName: \"kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp\") pod \"da821776-961c-4f2f-8be2-b817450eeb20\" (UID: \"da821776-961c-4f2f-8be2-b817450eeb20\") "
Sep 18 22:44:21.411867 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.411840    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da821776-961c-4f2f-8be2-b817450eeb20" (UID: "da821776-961c-4f2f-8be2-b817450eeb20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:44:21.411970 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.411879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp" (OuterVolumeSpecName: "kube-api-access-gf2vp") pod "da821776-961c-4f2f-8be2-b817450eeb20" (UID: "da821776-961c-4f2f-8be2-b817450eeb20"). InnerVolumeSpecName "kube-api-access-gf2vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:44:21.510713 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.510684    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2vp\" (UniqueName: \"kubernetes.io/projected/da821776-961c-4f2f-8be2-b817450eeb20-kube-api-access-gf2vp\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:44:21.510713 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.510714    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da821776-961c-4f2f-8be2-b817450eeb20-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:44:21.790279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.790205    2563 generic.go:356] "Generic (PLEG): container finished" podID="da821776-961c-4f2f-8be2-b817450eeb20" containerID="1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813" exitCode=0
Sep 18 22:44:21.790279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.790252    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"
Sep 18 22:44:21.790467 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.790283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" event={"ID":"da821776-961c-4f2f-8be2-b817450eeb20","Type":"ContainerDied","Data":"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"}
Sep 18 22:44:21.790467 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.790325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg" event={"ID":"da821776-961c-4f2f-8be2-b817450eeb20","Type":"ContainerDied","Data":"a9b1ce319b436a6ff24e756452b0447d2d4e16b59ad6a88c7268b2120ec17aa9"}
Sep 18 22:44:21.790467 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.790346    2563 scope.go:117] "RemoveContainer" containerID="1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"
Sep 18 22:44:21.798916 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.798899    2563 scope.go:117] "RemoveContainer" containerID="1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"
Sep 18 22:44:21.799176 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:21.799159    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813\": container with ID starting with 1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813 not found: ID does not exist" containerID="1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"
Sep 18 22:44:21.799225 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.799183    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813"} err="failed to get container status \"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813\": rpc error: code = NotFound desc = could not find container \"1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813\": container with ID starting with 1410916dda44d9c047c8fa6601eebf58d2b5aeffd8647704e28edbf490f4b813 not found: ID does not exist"
Sep 18 22:44:21.811282 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.811261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:44:21.817884 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.817864    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-74fc5hvchg"]
Sep 18 22:44:21.882610 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:21.882583    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da821776-961c-4f2f-8be2-b817450eeb20" path="/var/lib/kubelet/pods/da821776-961c-4f2f-8be2-b817450eeb20/volumes"
Sep 18 22:44:23.263052 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:23.263019    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:44:27.876441 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:27.876390    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:44:27.876907 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:27.876665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:44:31.158324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:31.158286    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:34.108613 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.108579    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:44:34.109062 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.109034    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main"
Sep 18 22:44:34.109062 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.109049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main"
Sep 18 22:44:34.109177 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.109142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da821776-961c-4f2f-8be2-b817450eeb20" containerName="main"
Sep 18 22:44:34.113684 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.113662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.116721 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.116699    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 22:44:34.116854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.116699    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 22:44:34.116854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.116740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 22:44:34.116979 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.116774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-v57kr\""
Sep 18 22:44:34.122682 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.122661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:44:34.204840 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.204840 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.205053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.205053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.205053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96rw\" (UniqueName: \"kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.205053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.204978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306006 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.305970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306192 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96rw\" (UniqueName: \"kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306603 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306777 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.306847 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.306756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.308599 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.308579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.308736 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.308720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.314326 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.314306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96rw\" (UniqueName: \"kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.425796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.425736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:34.707251 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.707173    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:44:34.710234 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:44:34.710198    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52440d19_f37a_4ab5_b4d6_a60d37fe7afd.slice/crio-e3b2d1fa0ea3b2449d5e450054fe020a92ad41f64c962619982195a987606b75 WatchSource:0}: Error finding container e3b2d1fa0ea3b2449d5e450054fe020a92ad41f64c962619982195a987606b75: Status 404 returned error can't find the container with id e3b2d1fa0ea3b2449d5e450054fe020a92ad41f64c962619982195a987606b75
Sep 18 22:44:34.849632 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.849597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" event={"ID":"52440d19-f37a-4ab5-b4d6-a60d37fe7afd","Type":"ContainerStarted","Data":"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"}
Sep 18 22:44:34.849827 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.849653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" event={"ID":"52440d19-f37a-4ab5-b4d6-a60d37fe7afd","Type":"ContainerStarted","Data":"e3b2d1fa0ea3b2449d5e450054fe020a92ad41f64c962619982195a987606b75"}
Sep 18 22:44:34.871128 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:34.871085    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" podStartSLOduration=0.871070989 podStartE2EDuration="871.070989ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:44:34.869464005 +0000 UTC m=+3215.068172303" watchObservedRunningTime="2026-09-18 22:44:34.871070989 +0000 UTC m=+3215.069779325"
Sep 18 22:44:35.426649 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:35.426608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:38.291317 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:38.291281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main" containerID="cri-o://d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8" gracePeriod=45
Sep 18 22:44:41.159442 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:41.159383    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:42.876822 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:42.876786    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:44:42.877272 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:42.877006    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:44:44.426561 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:44.426527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:44.438489 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:44.438464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:44.903561 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:44.903532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:44:51.158212 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:51.158173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:44:51.158636 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:51.158301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:44:54.876692 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:44:54.876663    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:44:54.877107 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:44:54.876888    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:01.158793 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:01.158755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:45:05.877085 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:05.877053    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:45:05.988952 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:05.988933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:45:05.989407 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:05.989381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:45:05.989538 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:05.989514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerStarted","Data":"1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"}
Sep 18 22:45:05.989823 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:05.989807    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:05.990043 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:05.990023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:06.995196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.995168    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:45:06.995701 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.995683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/11.log"
Sep 18 22:45:06.995745 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.995719    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b" exitCode=1
Sep 18 22:45:06.995812 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.995794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"}
Sep 18 22:45:06.995863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.995852    2563 scope.go:117] "RemoveContainer" containerID="f23b20686e06aba4b37e8e46ce02fea392cbc4d8517fd84c01f2c0bd5361e832"
Sep 18 22:45:06.996114 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:06.996101    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:06.996298 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:06.996281    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:08.002696 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.002656    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:45:08.539058 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.539035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:45:08.697380 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.697351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs\") pod \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") "
Sep 18 22:45:08.697547 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.697441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r6rz\" (UniqueName: \"kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz\") pod \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\" (UID: \"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e\") "
Sep 18 22:45:08.699573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.699548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz" (OuterVolumeSpecName: "kube-api-access-8r6rz") pod "9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" (UID: "9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e"). InnerVolumeSpecName "kube-api-access-8r6rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:08.699664 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.699586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" (UID: "9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:45:08.798073 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.798042    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6rz\" (UniqueName: \"kubernetes.io/projected/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-kube-api-access-8r6rz\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:08.798073 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:08.798071    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:09.008425 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.008379    2563 generic.go:356] "Generic (PLEG): container finished" podID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerID="d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8" exitCode=0
Sep 18 22:45:09.008791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.008439    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"
Sep 18 22:45:09.008791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.008436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" event={"ID":"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e","Type":"ContainerDied","Data":"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"}
Sep 18 22:45:09.008791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.008498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t" event={"ID":"9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e","Type":"ContainerDied","Data":"cd52c39f3552a587c3f7dcc6689091f6d6cfb29d8a2983f207bd70f8756ac179"}
Sep 18 22:45:09.008791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.008537    2563 scope.go:117] "RemoveContainer" containerID="d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"
Sep 18 22:45:09.017663 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.017646    2563 scope.go:117] "RemoveContainer" containerID="d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"
Sep 18 22:45:09.017943 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:09.017919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8\": container with ID starting with d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8 not found: ID does not exist" containerID="d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"
Sep 18 22:45:09.018002 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.017941    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8"} err="failed to get container status \"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8\": rpc error: code = NotFound desc = could not find container \"d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8\": container with ID starting with d84ba715803522fe82497ad60f80926a4bbe0eb83f59cb0eb3fcde714a30cec8 not found: ID does not exist"
Sep 18 22:45:09.030551 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.030528    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:45:09.035607 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.035585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5dc87bb622b7t"]
Sep 18 22:45:09.455857 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.455821    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:45:09.467002 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:09.466955    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:45:09.467002 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:45:09.467002 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:45:09.467002 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336" cmd=["/bin/sleep","15"]
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.467020    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" message="" err=<
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:09.467059    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerName="main" containerID="cri-o://2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"
Sep 18 22:45:09.467331 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.467100    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerName="main" containerID="cri-o://2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336" gracePeriod=60
Sep 18 22:45:09.814540 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.814519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:45:09.882389 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.882350    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" path="/var/lib/kubelet/pods/9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e/volumes"
Sep 18 22:45:09.908134 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908261 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908261 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908363 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908344    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k96rw\" (UniqueName: \"kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw\") pod \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\" (UID: \"52440d19-f37a-4ab5-b4d6-a60d37fe7afd\") "
Sep 18 22:45:09.908547 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908522    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache" (OuterVolumeSpecName: "model-cache") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:09.908602 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home" (OuterVolumeSpecName: "home") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:09.908655 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908625    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:09.908655 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908644    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:09.908783 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.908764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:09.910724 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.910689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw" (OuterVolumeSpecName: "kube-api-access-k96rw") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "kube-api-access-k96rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:09.910855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.910765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:45:09.910855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:09.910773    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm" (OuterVolumeSpecName: "dshm") pod "52440d19-f37a-4ab5-b4d6-a60d37fe7afd" (UID: "52440d19-f37a-4ab5-b4d6-a60d37fe7afd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:10.009139 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.009066    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:10.009139 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.009110    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96rw\" (UniqueName: \"kubernetes.io/projected/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-kube-api-access-k96rw\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:10.009139 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.009134    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:10.009519 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.009154    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52440d19-f37a-4ab5-b4d6-a60d37fe7afd-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:45:10.015469 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.015445    2563 generic.go:356] "Generic (PLEG): container finished" podID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerID="2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336" exitCode=0
Sep 18 22:45:10.015591 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.015489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" event={"ID":"52440d19-f37a-4ab5-b4d6-a60d37fe7afd","Type":"ContainerDied","Data":"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"}
Sep 18 22:45:10.015591 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.015538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54" event={"ID":"52440d19-f37a-4ab5-b4d6-a60d37fe7afd","Type":"ContainerDied","Data":"e3b2d1fa0ea3b2449d5e450054fe020a92ad41f64c962619982195a987606b75"}
Sep 18 22:45:10.015591 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.015493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"
Sep 18 22:45:10.015747 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.015568    2563 scope.go:117] "RemoveContainer" containerID="2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"
Sep 18 22:45:10.024873 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.024857    2563 scope.go:117] "RemoveContainer" containerID="2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"
Sep 18 22:45:10.025126 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:10.025107    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336\": container with ID starting with 2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336 not found: ID does not exist" containerID="2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"
Sep 18 22:45:10.025171 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.025133    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336"} err="failed to get container status \"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336\": rpc error: code = NotFound desc = could not find container \"2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336\": container with ID starting with 2469b89be4699c0226817960d647c05f35475b09680be6258cd4080143203336 not found: ID does not exist"
Sep 18 22:45:10.037282 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.037260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:45:10.041050 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:10.041030    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ffj54"]
Sep 18 22:45:11.883088 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:11.883055    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" path="/var/lib/kubelet/pods/52440d19-f37a-4ab5-b4d6-a60d37fe7afd/volumes"
Sep 18 22:45:13.586208 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586180    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:45:13.586541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586494    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main"
Sep 18 22:45:13.586541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586505    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main"
Sep 18 22:45:13.586541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586523    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerName="main"
Sep 18 22:45:13.586541 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586529    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerName="main"
Sep 18 22:45:13.586672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586594    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52440d19-f37a-4ab5-b4d6-a60d37fe7afd" containerName="main"
Sep 18 22:45:13.586672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.586606    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbc5d5e-2f1d-4d6a-9b15-663caa45c63e" containerName="main"
Sep 18 22:45:13.590980 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.590958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.594129 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.594105    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:45:13.594129 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.594121    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 22:45:13.594641 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.594624    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-6mnmr\""
Sep 18 22:45:13.596806 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.596780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 22:45:13.612560 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.612532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:45:13.739278 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739473 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739473 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4k7w\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739473 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.739764 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.739656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840635 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840794 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840794 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840794 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840916 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.840916 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4k7w\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841023 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841023 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841023 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.840981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841285 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.841256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841362 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.841337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841469 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.841437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841586 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.841449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.841586 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.841574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.842982 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.842950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.843232 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.843214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.848023 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.847999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4k7w\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.848412 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.848380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token\") pod \"router-gateway-1-openshift-default-66c8c97dfb-mjhqm\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:13.901732 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:13.901708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:14.182779 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:14.182755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:45:14.184929 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:45:14.184902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80e2640_aa9d_4d2c_9d2c_136106cafbae.slice/crio-9b8cbf1d1bb8ce666d5326450656a90baff8dbf735d3cb2fe31bb22df306b640 WatchSource:0}: Error finding container 9b8cbf1d1bb8ce666d5326450656a90baff8dbf735d3cb2fe31bb22df306b640: Status 404 returned error can't find the container with id 9b8cbf1d1bb8ce666d5326450656a90baff8dbf735d3cb2fe31bb22df306b640
Sep 18 22:45:14.187036 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:14.187018    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:45:14.301854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:14.301827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:45:14.302175 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:14.302162    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:14.302335 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:14.302318    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:14.964411 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:14.964368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:45:15.039830 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:15.039792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerStarted","Data":"9b8cbf1d1bb8ce666d5326450656a90baff8dbf735d3cb2fe31bb22df306b640"}
Sep 18 22:45:15.040113 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:15.040096    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:15.040288 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:15.040266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:16.894256 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:16.894213    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:16.894583 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:16.894291    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:16.894583 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:16.894324    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:17.053410 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:17.053355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerStarted","Data":"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"}
Sep 18 22:45:17.076795 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:17.076720    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podStartSLOduration=1.3698638060000001 podStartE2EDuration="4.076701898s" totalImagesPullingTime="2.706838092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:17.075856973 +0000 UTC m=+3257.274565273" watchObservedRunningTime="2026-09-18 22:45:17.076701898 +0000 UTC m=+3257.275410198"
Sep 18 22:45:17.902474 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:17.902440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:17.907433 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:17.907389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:18.058035 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:18.058005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:18.058945 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:18.058922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:28.108572 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.108543    2563 generic.go:356] "Generic (PLEG): container finished" podID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerID="3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632" exitCode=0
Sep 18 22:45:28.108973 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.108605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerDied","Data":"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"}
Sep 18 22:45:28.108973 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.108928    2563 scope.go:117] "RemoveContainer" containerID="3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"
Sep 18 22:45:28.110838 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.110800    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:28.110930 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.110895    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:28.110971 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:28.110946    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:45:29.115318 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:29.115285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerStarted","Data":"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"}
Sep 18 22:45:29.877076 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:29.877037    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:29.877293 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:29.877269    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:29.902638 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:29.902610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:29.903668 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:29.903639    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 18 22:45:30.902796 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:30.902745    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 18 22:45:31.902143 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:31.902108    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 18 22:45:32.902539 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:32.902500    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 18 22:45:33.058097 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:33.058052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:33.902534 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:33.902495    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 18 22:45:34.905932 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:34.905903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:34.906840 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:34.906820    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:45:38.685714 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.685682    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:45:38.688875 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.688852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.692151 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.692124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-bz6sv\""
Sep 18 22:45:38.692151 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.692136    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-pm4c9\""
Sep 18 22:45:38.692332 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.692223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 22:45:38.700345 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.700323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:45:38.752227 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.752201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.752426 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.752298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vsz\" (UniqueName: \"kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.853608 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.853566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.853813 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.853625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vsz\" (UniqueName: \"kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.856229 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.856200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:38.861965 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:38.861934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vsz\" (UniqueName: \"kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz\") pod \"router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:39.000715 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:39.000612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:39.277154 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:39.277128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:45:39.279072 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:45:39.279042    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c47b58_8d02_4902_a536_c29b00e3e357.slice/crio-7ec6163e62d44f33d9da910304882be30c700f24ee94cca3563133f7e0e4000c WatchSource:0}: Error finding container 7ec6163e62d44f33d9da910304882be30c700f24ee94cca3563133f7e0e4000c: Status 404 returned error can't find the container with id 7ec6163e62d44f33d9da910304882be30c700f24ee94cca3563133f7e0e4000c
Sep 18 22:45:40.166272 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:40.166239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" event={"ID":"49c47b58-8d02-4902-a536-c29b00e3e357","Type":"ContainerStarted","Data":"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"}
Sep 18 22:45:40.166676 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:40.166287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" event={"ID":"49c47b58-8d02-4902-a536-c29b00e3e357","Type":"ContainerStarted","Data":"7ec6163e62d44f33d9da910304882be30c700f24ee94cca3563133f7e0e4000c"}
Sep 18 22:45:40.166676 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:40.166341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:45:40.189135 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:40.189076    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podStartSLOduration=2.189059961 podStartE2EDuration="2.189059961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:40.18872829 +0000 UTC m=+3280.387436588" watchObservedRunningTime="2026-09-18 22:45:40.189059961 +0000 UTC m=+3280.387768263"
Sep 18 22:45:42.876579 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:42.876546    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:42.876968 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:42.876715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:45:53.876634 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:45:53.876603    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:45:53.877041 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:45:53.876780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:46:02.077440 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:02.077416    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:46:02.088243 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:02.088215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:46:02.091602 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:02.091580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:46:02.101722 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:02.101704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:46:07.876874 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:07.876839    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:46:07.877267 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:46:07.877069    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:46:11.172627 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:11.172595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:46:21.813728 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.813646    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:46:21.818648 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.818624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.821196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.821174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-pcgsz\""
Sep 18 22:46:21.822017 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.821993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 22:46:21.822148 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.822026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 22:46:21.822148 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.822124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 22:46:21.826246 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.825985    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:46:21.878860 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.878839    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:46:21.879014 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:46:21.878995    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:46:21.973674 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.973829 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ks7\" (UniqueName: \"kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.973829 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.973829 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.974021 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:21.974021 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:21.973893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.074830 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.074733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ks7\" (UniqueName: \"kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.074830 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.074786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075063 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.074898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075063 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.074998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075198 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.075058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075198 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.075142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075325 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.075289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075546 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.075517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.075672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.075543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.077364 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.077342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.077561 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.077542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.082673 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.082650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ks7\" (UniqueName: \"kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.131362 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.131337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:22.406338 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:22.406307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:46:22.409829 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:46:22.409801    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a7c412_ee15_43ad_b32a_63429d99b686.slice/crio-7670c401e1aaed8ea5ff60f5c723d9764e8ddd9cc567f98b60123ad44121613b WatchSource:0}: Error finding container 7670c401e1aaed8ea5ff60f5c723d9764e8ddd9cc567f98b60123ad44121613b: Status 404 returned error can't find the container with id 7670c401e1aaed8ea5ff60f5c723d9764e8ddd9cc567f98b60123ad44121613b
Sep 18 22:46:23.359940 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:23.359905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" event={"ID":"d4a7c412-ee15-43ad-b32a-63429d99b686","Type":"ContainerStarted","Data":"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"}
Sep 18 22:46:23.360332 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:23.359962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" event={"ID":"d4a7c412-ee15-43ad-b32a-63429d99b686","Type":"ContainerStarted","Data":"7670c401e1aaed8ea5ff60f5c723d9764e8ddd9cc567f98b60123ad44121613b"}
Sep 18 22:46:23.398331 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:23.398271    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" podStartSLOduration=2.398252627 podStartE2EDuration="2.398252627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:46:23.398013081 +0000 UTC m=+3323.596721379" watchObservedRunningTime="2026-09-18 22:46:23.398252627 +0000 UTC m=+3323.596960933"
Sep 18 22:46:24.131689 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:24.131656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:32.131540 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:32.131509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:32.143783 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:32.143760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:32.412457 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:32.412371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:46:33.876724 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:33.876696    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:46:33.877191 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:46:33.876845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:46:44.877211 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:44.877178    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:46:44.877584 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:46:44.877348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:46:55.877287 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:46:55.877247    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:46:55.877808 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:46:55.877558    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:47:07.882793 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:07.882763    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:47:07.883176 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:47:07.882954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:47:11.528298 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.528261    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:47:11.531428 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.531386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.534892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.534863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-hls7k\""
Sep 18 22:47:11.535037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.535011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 22:47:11.541072 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.541046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:47:11.669380 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.669344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p8d\" (UniqueName: \"kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.669614 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.669416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.770279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.770242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.770478 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.770325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p8d\" (UniqueName: \"kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.773005 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.772981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.779264 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.779198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p8d\" (UniqueName: \"kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:11.844247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:11.844210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:12.123608 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:12.123580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:47:12.125943 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:47:12.125909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca45f22_afed_4079_a485_ece1697a784c.slice/crio-3c10862d482138e7519311e11f8819fe7b57b603eb2ecdf5a151e8a666bd1eef WatchSource:0}: Error finding container 3c10862d482138e7519311e11f8819fe7b57b603eb2ecdf5a151e8a666bd1eef: Status 404 returned error can't find the container with id 3c10862d482138e7519311e11f8819fe7b57b603eb2ecdf5a151e8a666bd1eef
Sep 18 22:47:12.590002 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:12.589969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" event={"ID":"aca45f22-afed-4079-a485-ece1697a784c","Type":"ContainerStarted","Data":"57cc90646885088c383b8186edc4e046a879473d7fcb414cd9a4fa933f0c92f4"}
Sep 18 22:47:12.590409 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:12.590023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" event={"ID":"aca45f22-afed-4079-a485-ece1697a784c","Type":"ContainerStarted","Data":"3c10862d482138e7519311e11f8819fe7b57b603eb2ecdf5a151e8a666bd1eef"}
Sep 18 22:47:12.590409 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:12.590095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:12.608439 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:12.608369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podStartSLOduration=1.608355075 podStartE2EDuration="1.608355075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:47:12.607687787 +0000 UTC m=+3372.806396102" watchObservedRunningTime="2026-09-18 22:47:12.608355075 +0000 UTC m=+3372.807063374"
Sep 18 22:47:18.877332 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:18.877301    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:47:18.877717 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:47:18.877478    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:47:31.878871 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:31.878843    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:47:31.879362 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:47:31.879020    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:47:43.598650 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:43.598620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:47:44.876829 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:44.876751    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:47:44.877204 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:47:44.876907    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:47:48.144025 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:48.143979    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:47:51.172345 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:51.172307    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:47:53.730278 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:53.730243    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:47:57.877178 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:47:57.877147    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:47:57.877594 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:47:57.877345    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:48:01.172360 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:01.172325    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:03.164251 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:03.164212    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" containerID="cri-o://f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f" gracePeriod=45
Sep 18 22:48:03.598615 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:03.598585    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:08.753211 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:08.753174    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main" containerID="cri-o://57cc90646885088c383b8186edc4e046a879473d7fcb414cd9a4fa933f0c92f4" gracePeriod=45
Sep 18 22:48:11.172632 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:11.172595    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:11.173012 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:11.172690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:48:11.878612 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:11.878584    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:48:11.878815 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:11.878762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:48:13.598602 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:13.598568    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:21.171982 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:21.171946    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:22.877242 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:22.877213    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:48:22.877632 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:22.877365    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:48:23.598567 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:23.598531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:23.598730 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:23.598626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:48:31.172515 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:31.172471    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:33.428740 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.428685    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:48:33.520465 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.520441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs\") pod \"49c47b58-8d02-4902-a536-c29b00e3e357\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") "
Sep 18 22:48:33.520618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.520498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5vsz\" (UniqueName: \"kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz\") pod \"49c47b58-8d02-4902-a536-c29b00e3e357\" (UID: \"49c47b58-8d02-4902-a536-c29b00e3e357\") "
Sep 18 22:48:33.522757 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.522725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz" (OuterVolumeSpecName: "kube-api-access-w5vsz") pod "49c47b58-8d02-4902-a536-c29b00e3e357" (UID: "49c47b58-8d02-4902-a536-c29b00e3e357"). InnerVolumeSpecName "kube-api-access-w5vsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:48:33.522861 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.522755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49c47b58-8d02-4902-a536-c29b00e3e357" (UID: "49c47b58-8d02-4902-a536-c29b00e3e357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:48:33.599062 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.599037    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:48:33.621495 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.621471    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49c47b58-8d02-4902-a536-c29b00e3e357-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:33.621495 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.621492    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5vsz\" (UniqueName: \"kubernetes.io/projected/49c47b58-8d02-4902-a536-c29b00e3e357-kube-api-access-w5vsz\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:33.962128 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.962098    2563 generic.go:356] "Generic (PLEG): container finished" podID="49c47b58-8d02-4902-a536-c29b00e3e357" containerID="f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f" exitCode=0
Sep 18 22:48:33.962304 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.962143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"
Sep 18 22:48:33.962304 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.962179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" event={"ID":"49c47b58-8d02-4902-a536-c29b00e3e357","Type":"ContainerDied","Data":"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"}
Sep 18 22:48:33.962304 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.962228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4" event={"ID":"49c47b58-8d02-4902-a536-c29b00e3e357","Type":"ContainerDied","Data":"7ec6163e62d44f33d9da910304882be30c700f24ee94cca3563133f7e0e4000c"}
Sep 18 22:48:33.962304 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.962251    2563 scope.go:117] "RemoveContainer" containerID="f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"
Sep 18 22:48:33.970910 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.970893    2563 scope.go:117] "RemoveContainer" containerID="f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"
Sep 18 22:48:33.971157 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:33.971140    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f\": container with ID starting with f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f not found: ID does not exist" containerID="f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"
Sep 18 22:48:33.971221 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.971162    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f"} err="failed to get container status \"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f\": rpc error: code = NotFound desc = could not find container \"f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f\": container with ID starting with f1faf9fbd5ad982418d03a2dfaa043f67f9f902b2dd9b2fed7b19922c23a9a2f not found: ID does not exist"
Sep 18 22:48:33.978296 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.978278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:48:33.982242 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:33.982219    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-69597ff9d-pz6f4"]
Sep 18 22:48:35.883429 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:35.883376    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" path="/var/lib/kubelet/pods/49c47b58-8d02-4902-a536-c29b00e3e357/volumes"
Sep 18 22:48:36.877122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:36.877094    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:48:36.877283 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:36.877261    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:48:38.988476 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:38.988446    2563 generic.go:356] "Generic (PLEG): container finished" podID="aca45f22-afed-4079-a485-ece1697a784c" containerID="57cc90646885088c383b8186edc4e046a879473d7fcb414cd9a4fa933f0c92f4" exitCode=0
Sep 18 22:48:38.988832 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:38.988521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" event={"ID":"aca45f22-afed-4079-a485-ece1697a784c","Type":"ContainerDied","Data":"57cc90646885088c383b8186edc4e046a879473d7fcb414cd9a4fa933f0c92f4"}
Sep 18 22:48:39.003996 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.003973    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:48:39.061666 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.061607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49p8d\" (UniqueName: \"kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d\") pod \"aca45f22-afed-4079-a485-ece1697a784c\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") "
Sep 18 22:48:39.061772 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.061676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs\") pod \"aca45f22-afed-4079-a485-ece1697a784c\" (UID: \"aca45f22-afed-4079-a485-ece1697a784c\") "
Sep 18 22:48:39.063619 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.063590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aca45f22-afed-4079-a485-ece1697a784c" (UID: "aca45f22-afed-4079-a485-ece1697a784c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:48:39.063721 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.063611    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d" (OuterVolumeSpecName: "kube-api-access-49p8d") pod "aca45f22-afed-4079-a485-ece1697a784c" (UID: "aca45f22-afed-4079-a485-ece1697a784c"). InnerVolumeSpecName "kube-api-access-49p8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:48:39.162573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.162550    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca45f22-afed-4079-a485-ece1697a784c-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:39.162573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.162572    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49p8d\" (UniqueName: \"kubernetes.io/projected/aca45f22-afed-4079-a485-ece1697a784c-kube-api-access-49p8d\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:39.993955 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.993931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"
Sep 18 22:48:39.993955 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.993940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r" event={"ID":"aca45f22-afed-4079-a485-ece1697a784c","Type":"ContainerDied","Data":"3c10862d482138e7519311e11f8819fe7b57b603eb2ecdf5a151e8a666bd1eef"}
Sep 18 22:48:39.994364 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:39.993992    2563 scope.go:117] "RemoveContainer" containerID="57cc90646885088c383b8186edc4e046a879473d7fcb414cd9a4fa933f0c92f4"
Sep 18 22:48:40.010077 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:40.010053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:48:40.015829 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:40.015808    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schepfk2r"]
Sep 18 22:48:41.267739 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.267709    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:48:41.268074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main"
Sep 18 22:48:41.268074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main"
Sep 18 22:48:41.268074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main"
Sep 18 22:48:41.268074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main"
Sep 18 22:48:41.268219 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268129    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c47b58-8d02-4902-a536-c29b00e3e357" containerName="main"
Sep 18 22:48:41.268219 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.268142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca45f22-afed-4079-a485-ece1697a784c" containerName="main"
Sep 18 22:48:41.272414 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.272377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.276212 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.276169    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 22:48:41.276744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.276262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-9dgjn\""
Sep 18 22:48:41.276744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.276475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 22:48:41.276744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.276263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-kmt8t\""
Sep 18 22:48:41.276744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.276294    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 22:48:41.282851 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.282830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:48:41.378113 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.378084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhx7\" (UniqueName: \"kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.378275 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.378191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.453296 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.453264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:48:41.453479 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.453458    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy" containerID="cri-o://fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635" gracePeriod=30
Sep 18 22:48:41.478821 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.478793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.479034 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.478875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhx7\" (UniqueName: \"kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.482200 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.482176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.490944 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.490922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhx7\" (UniqueName: \"kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.587042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.587007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:41.861196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.861125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:48:41.863581 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:48:41.863544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b4de53_40ea_44a5_a507_44023a88fc6b.slice/crio-8fed505f79000c2a690b6330484d8753ae5314d21bd87caa87ca236baa43fd4d WatchSource:0}: Error finding container 8fed505f79000c2a690b6330484d8753ae5314d21bd87caa87ca236baa43fd4d: Status 404 returned error can't find the container with id 8fed505f79000c2a690b6330484d8753ae5314d21bd87caa87ca236baa43fd4d
Sep 18 22:48:41.884753 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:41.884730    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca45f22-afed-4079-a485-ece1697a784c" path="/var/lib/kubelet/pods/aca45f22-afed-4079-a485-ece1697a784c/volumes"
Sep 18 22:48:42.008268 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:42.008231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" event={"ID":"a4b4de53-40ea-44a5-a507-44023a88fc6b","Type":"ContainerStarted","Data":"ec32f8f54de79d3470b1a3b4df7ccd9da5031b8d8b9a9945d9082c003c18c73d"}
Sep 18 22:48:42.008492 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:42.008291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" event={"ID":"a4b4de53-40ea-44a5-a507-44023a88fc6b","Type":"ContainerStarted","Data":"8fed505f79000c2a690b6330484d8753ae5314d21bd87caa87ca236baa43fd4d"}
Sep 18 22:48:42.008492 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:42.008360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:48:42.028006 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:42.027953    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podStartSLOduration=1.027939012 podStartE2EDuration="1.027939012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:48:42.025721257 +0000 UTC m=+3462.224429556" watchObservedRunningTime="2026-09-18 22:48:42.027939012 +0000 UTC m=+3462.226647310"
Sep 18 22:48:46.696224 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.696198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:48:46.724589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4k7w\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.724900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.724830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket\") pod \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\" (UID: \"e80e2640-aa9d-4d2c-9d2c-136106cafbae\") "
Sep 18 22:48:46.725341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.725244    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:46.725341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.725268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:46.725341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.725286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:46.725569 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.725344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data" (OuterVolumeSpecName: "istio-data") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:46.727517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.727482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:46.727981 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.727945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w" (OuterVolumeSpecName: "kube-api-access-m4k7w") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "kube-api-access-m4k7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:48:46.728336 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.728311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:48:46.728870 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.728844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:48:46.729892 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.729867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token" (OuterVolumeSpecName: "istio-token") pod "e80e2640-aa9d-4d2c-9d2c-136106cafbae" (UID: "e80e2640-aa9d-4d2c-9d2c-136106cafbae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:48:46.825691 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825661    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-socket\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825691 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825686    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-podinfo\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825691 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825696    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istiod-ca-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825705    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-credential-socket\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825714    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-data\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825723    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-envoy\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825730    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4k7w\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-kube-api-access-m4k7w\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825739    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80e2640-aa9d-4d2c-9d2c-136106cafbae-workload-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:46.825869 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:46.825747    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80e2640-aa9d-4d2c-9d2c-136106cafbae-istio-token\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:48:47.032489 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.032421    2563 generic.go:356] "Generic (PLEG): container finished" podID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerID="fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635" exitCode=0
Sep 18 22:48:47.032489 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.032458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerDied","Data":"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"}
Sep 18 22:48:47.032489 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.032470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"
Sep 18 22:48:47.032698 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.032501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm" event={"ID":"e80e2640-aa9d-4d2c-9d2c-136106cafbae","Type":"ContainerDied","Data":"9b8cbf1d1bb8ce666d5326450656a90baff8dbf735d3cb2fe31bb22df306b640"}
Sep 18 22:48:47.032698 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.032538    2563 scope.go:117] "RemoveContainer" containerID="fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"
Sep 18 22:48:47.047753 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.047730    2563 scope.go:117] "RemoveContainer" containerID="3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"
Sep 18 22:48:47.057063 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.057045    2563 scope.go:117] "RemoveContainer" containerID="fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"
Sep 18 22:48:47.057323 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:47.057306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635\": container with ID starting with fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635 not found: ID does not exist" containerID="fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"
Sep 18 22:48:47.057369 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.057330    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635"} err="failed to get container status \"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635\": rpc error: code = NotFound desc = could not find container \"fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635\": container with ID starting with fd97ee5f35b974ecd584ce29d7862b52f90e5509ed72607b295db7996568d635 not found: ID does not exist"
Sep 18 22:48:47.057369 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.057345    2563 scope.go:117] "RemoveContainer" containerID="3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"
Sep 18 22:48:47.057620 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:47.057603    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632\": container with ID starting with 3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632 not found: ID does not exist" containerID="3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"
Sep 18 22:48:47.057671 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.057623    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632"} err="failed to get container status \"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632\": rpc error: code = NotFound desc = could not find container \"3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632\": container with ID starting with 3d773941ab20bc625110bce9b7f42ee92c1e187d05272227314f54f99cdfe632 not found: ID does not exist"
Sep 18 22:48:47.061348 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.061325    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:48:47.065154 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.065135    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-66c8c97dfb-mjhqm"]
Sep 18 22:48:47.877006 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.876978    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:48:47.877377 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:48:47.877187    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:48:47.882702 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:48:47.882680    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" path="/var/lib/kubelet/pods/e80e2640-aa9d-4d2c-9d2c-136106cafbae/volumes"
Sep 18 22:49:00.316247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.316215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:49:00.325073 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:00.324988    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:49:00.325073 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:49:00.325073 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:49:00.325073 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15" cmd=["/bin/sleep","15"]
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.325156    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" message="" err=<
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:00.325179    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main" containerID="cri-o://f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"
Sep 18 22:49:00.325454 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.325273    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main" containerID="cri-o://f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15" gracePeriod=60
Sep 18 22:49:00.401900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.401868    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 18 22:49:00.679319 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.679294    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:49:00.728628 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ks7\" (UniqueName: \"kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.728758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.728758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728708    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.728866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.728866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.728866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs\") pod \"d4a7c412-ee15-43ad-b32a-63429d99b686\" (UID: \"d4a7c412-ee15-43ad-b32a-63429d99b686\") "
Sep 18 22:49:00.729079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.728912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home" (OuterVolumeSpecName: "home") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:00.729079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.729014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:00.729079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.729031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:00.729238 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.729144    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:00.729238 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.729161    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:00.729238 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.729172    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:00.730704 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.730681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7" (OuterVolumeSpecName: "kube-api-access-b2ks7") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "kube-api-access-b2ks7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:49:00.730836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.730818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm" (OuterVolumeSpecName: "dshm") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:00.730910 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.730838    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4a7c412-ee15-43ad-b32a-63429d99b686" (UID: "d4a7c412-ee15-43ad-b32a-63429d99b686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:49:00.830137 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.830118    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4a7c412-ee15-43ad-b32a-63429d99b686-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:00.830137 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.830136    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4a7c412-ee15-43ad-b32a-63429d99b686-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:00.830269 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:00.830146    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ks7\" (UniqueName: \"kubernetes.io/projected/d4a7c412-ee15-43ad-b32a-63429d99b686-kube-api-access-b2ks7\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:49:01.104918 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.104888    2563 generic.go:356] "Generic (PLEG): container finished" podID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerID="f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15" exitCode=0
Sep 18 22:49:01.105074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.104931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"
Sep 18 22:49:01.105074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.104943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" event={"ID":"d4a7c412-ee15-43ad-b32a-63429d99b686","Type":"ContainerDied","Data":"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"}
Sep 18 22:49:01.105074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.104987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl" event={"ID":"d4a7c412-ee15-43ad-b32a-63429d99b686","Type":"ContainerDied","Data":"7670c401e1aaed8ea5ff60f5c723d9764e8ddd9cc567f98b60123ad44121613b"}
Sep 18 22:49:01.105074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.105009    2563 scope.go:117] "RemoveContainer" containerID="f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"
Sep 18 22:49:01.113744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.113720    2563 scope.go:117] "RemoveContainer" containerID="f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"
Sep 18 22:49:01.113981 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:01.113965    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15\": container with ID starting with f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15 not found: ID does not exist" containerID="f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"
Sep 18 22:49:01.114044 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.113988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15"} err="failed to get container status \"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15\": rpc error: code = NotFound desc = could not find container \"f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15\": container with ID starting with f7a18acdcaf5de79f390911ddfb8c307f13619ab5fc704274f9ecf1d34eb1a15 not found: ID does not exist"
Sep 18 22:49:01.125132 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.125109    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:49:01.131230 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.131209    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-kv7bl"]
Sep 18 22:49:01.882258 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:01.882226    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" path="/var/lib/kubelet/pods/d4a7c412-ee15-43ad-b32a-63429d99b686/volumes"
Sep 18 22:49:02.876854 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:02.876825    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:49:02.877024 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:02.877007    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:49:13.014382 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:13.014349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:49:17.877424 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:17.877334    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:49:17.877776 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:17.877543    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:49:31.878596 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:31.878564    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:49:31.879006 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:31.878755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:49:46.877127 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:49:46.877095    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:49:46.877622 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:49:46.877267    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:01.878090 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:01.878053    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:50:01.878510 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:01.878300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:04.510742 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.510705    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:50:04.511201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511180    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.511254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511205    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.511254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511227    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.511254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511236    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.511374 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511261    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main"
Sep 18 22:50:04.511374 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main"
Sep 18 22:50:04.511374 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511339    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4a7c412-ee15-43ad-b32a-63429d99b686" containerName="main"
Sep 18 22:50:04.511374 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511352    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.511374 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.511363    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80e2640-aa9d-4d2c-9d2c-136106cafbae" containerName="istio-proxy"
Sep 18 22:50:04.515050 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.515031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.517749 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.517717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 22:50:04.518556 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.518536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-rbr4p\""
Sep 18 22:50:04.518665 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.518578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 22:50:04.518665 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.518577    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 22:50:04.525131 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.525108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:50:04.599227 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.599361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.599361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.599361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.599361 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.599530 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.599370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7q8\" (UniqueName: \"kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700656 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7q8\" (UniqueName: \"kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.700853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.700785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.701211 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.701178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.701413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.701224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.701517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.701251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.703286 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.703258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.703442 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.703416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.715745 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.715719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7q8\" (UniqueName: \"kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.720653 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.720624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:50:04.724134 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.724119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.726466 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.726445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-8rrh9\""
Sep 18 22:50:04.737277 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.737257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:50:04.801642 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.801580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.801642 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.801623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5mt\" (UniqueName: \"kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.826559 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.826531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:04.902639 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.902179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.902639 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.902245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5mt\" (UniqueName: \"kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.905257 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.905235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:04.911368 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:04.911345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5mt\" (UniqueName: \"kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:05.037989 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.037964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:05.104672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.104647    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:50:05.107889 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:50:05.107867    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580e659b_8f6a_4a84_8668_98690183816e.slice/crio-7fdea2bfa83c2ad374ad41fad3a31460532c5e902c92f427ed50fb6250804eb1 WatchSource:0}: Error finding container 7fdea2bfa83c2ad374ad41fad3a31460532c5e902c92f427ed50fb6250804eb1: Status 404 returned error can't find the container with id 7fdea2bfa83c2ad374ad41fad3a31460532c5e902c92f427ed50fb6250804eb1
Sep 18 22:50:05.314187 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.314154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:50:05.316431 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:50:05.316405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b4d540_cf78_4303_a402_ccd7e7cb49ef.slice/crio-891f9b2047bc076042a0746d7e06cb7b6fe3372d432e5ee8765706aaba497045 WatchSource:0}: Error finding container 891f9b2047bc076042a0746d7e06cb7b6fe3372d432e5ee8765706aaba497045: Status 404 returned error can't find the container with id 891f9b2047bc076042a0746d7e06cb7b6fe3372d432e5ee8765706aaba497045
Sep 18 22:50:05.383045 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.383020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" event={"ID":"64b4d540-cf78-4303-a402-ccd7e7cb49ef","Type":"ContainerStarted","Data":"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"}
Sep 18 22:50:05.383169 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.383066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" event={"ID":"64b4d540-cf78-4303-a402-ccd7e7cb49ef","Type":"ContainerStarted","Data":"891f9b2047bc076042a0746d7e06cb7b6fe3372d432e5ee8765706aaba497045"}
Sep 18 22:50:05.383169 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.383113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:05.384780 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.384756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" event={"ID":"580e659b-8f6a-4a84-8668-98690183816e","Type":"ContainerStarted","Data":"f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035"}
Sep 18 22:50:05.384886 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.384803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" event={"ID":"580e659b-8f6a-4a84-8668-98690183816e","Type":"ContainerStarted","Data":"7fdea2bfa83c2ad374ad41fad3a31460532c5e902c92f427ed50fb6250804eb1"}
Sep 18 22:50:05.402746 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.402700    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podStartSLOduration=1.402683489 podStartE2EDuration="1.402683489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:50:05.401433434 +0000 UTC m=+3545.600141734" watchObservedRunningTime="2026-09-18 22:50:05.402683489 +0000 UTC m=+3545.601391787"
Sep 18 22:50:05.418012 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.417975    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" podStartSLOduration=1.417961995 podStartE2EDuration="1.417961995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:50:05.417186041 +0000 UTC m=+3545.615894341" watchObservedRunningTime="2026-09-18 22:50:05.417961995 +0000 UTC m=+3545.616670293"
Sep 18 22:50:05.827511 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:05.827475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:14.826922 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:14.826873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:14.839122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:14.839097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:15.442703 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:15.442674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:50:15.876501 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:15.876477    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:50:16.441677 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.441651    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/13.log"
Sep 18 22:50:16.442138 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.442121    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/12.log"
Sep 18 22:50:16.442196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.442163    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb" exitCode=1
Sep 18 22:50:16.442267 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.442245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"}
Sep 18 22:50:16.442321 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.442310    2563 scope.go:117] "RemoveContainer" containerID="1326846e24fdfcbe84cccd9bc5bc5d135966a10310289769ba74c90d01172b9b"
Sep 18 22:50:16.442699 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:16.442683    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:50:16.442897 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:16.442874    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:17.449211 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:17.449185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/13.log"
Sep 18 22:50:24.302039 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:24.302007    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:50:24.302516 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:24.302317    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:50:24.302564 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:24.302524    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:24.964372 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:24.964338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:50:24.964720 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:24.964703    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:50:24.964883 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:24.964865    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:36.392302 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:36.392271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:50:36.877122 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:36.877094    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:50:36.877331 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:36.877310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:50:48.876769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:50:48.876694    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:50:48.877154 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:50:48.876864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:01.878618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:01.878592    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:51:01.879108 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:51:01.878797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:02.107173 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:02.107147    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/13.log"
Sep 18 22:51:02.119053 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:02.119029    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:51:02.119417 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:02.119378    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/13.log"
Sep 18 22:51:02.138061 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:02.138004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:51:11.736259 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:11.736222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:51:13.014413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:13.014363    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:16.876970 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:16.876940    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:51:16.877348 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:51:16.877114    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:23.014328 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:23.014292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:26.756739 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:26.756699    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" containerID="cri-o://ec32f8f54de79d3470b1a3b4df7ccd9da5031b8d8b9a9945d9082c003c18c73d" gracePeriod=45
Sep 18 22:51:29.876725 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:29.876696    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:51:29.877099 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:51:29.876861    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:33.014161 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:33.014114    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:33.014717 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:33.014236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:51:40.876749 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:40.876713    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:51:40.877196 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:51:40.876923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:43.014644 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:43.014606    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:51.879012 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:51.878981    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:51:51.879501 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:51:51.879133    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:51:53.014623 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:53.014586    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:51:56.896466 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:56.896439    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerID="ec32f8f54de79d3470b1a3b4df7ccd9da5031b8d8b9a9945d9082c003c18c73d" exitCode=0
Sep 18 22:51:56.896789 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:56.896497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" event={"ID":"a4b4de53-40ea-44a5-a507-44023a88fc6b","Type":"ContainerDied","Data":"ec32f8f54de79d3470b1a3b4df7ccd9da5031b8d8b9a9945d9082c003c18c73d"}
Sep 18 22:51:57.004083 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.004059    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:51:57.162530 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.162504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbhx7\" (UniqueName: \"kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7\") pod \"a4b4de53-40ea-44a5-a507-44023a88fc6b\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") "
Sep 18 22:51:57.162672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.162592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs\") pod \"a4b4de53-40ea-44a5-a507-44023a88fc6b\" (UID: \"a4b4de53-40ea-44a5-a507-44023a88fc6b\") "
Sep 18 22:51:57.164731 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.164707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4b4de53-40ea-44a5-a507-44023a88fc6b" (UID: "a4b4de53-40ea-44a5-a507-44023a88fc6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:51:57.164861 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.164769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7" (OuterVolumeSpecName: "kube-api-access-qbhx7") pod "a4b4de53-40ea-44a5-a507-44023a88fc6b" (UID: "a4b4de53-40ea-44a5-a507-44023a88fc6b"). InnerVolumeSpecName "kube-api-access-qbhx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:57.263578 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.263547    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhx7\" (UniqueName: \"kubernetes.io/projected/a4b4de53-40ea-44a5-a507-44023a88fc6b-kube-api-access-qbhx7\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:51:57.263677 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.263578    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4de53-40ea-44a5-a507-44023a88fc6b-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:51:57.901893 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.901870    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"
Sep 18 22:51:57.902322 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.901893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc" event={"ID":"a4b4de53-40ea-44a5-a507-44023a88fc6b","Type":"ContainerDied","Data":"8fed505f79000c2a690b6330484d8753ae5314d21bd87caa87ca236baa43fd4d"}
Sep 18 22:51:57.902322 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.901941    2563 scope.go:117] "RemoveContainer" containerID="ec32f8f54de79d3470b1a3b4df7ccd9da5031b8d8b9a9945d9082c003c18c73d"
Sep 18 22:51:57.918048 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.918026    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:51:57.921808 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:57.921788    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebf8fc"]
Sep 18 22:51:59.882413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:51:59.882363    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" path="/var/lib/kubelet/pods/a4b4de53-40ea-44a5-a507-44023a88fc6b/volumes"
Sep 18 22:52:03.876329 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:03.876304    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:52:03.876767 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:03.876479    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:52:15.251216 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.251140    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:52:15.251594 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.251517    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main"
Sep 18 22:52:15.251594 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.251530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main"
Sep 18 22:52:15.251594 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.251596    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b4de53-40ea-44a5-a507-44023a88fc6b" containerName="main"
Sep 18 22:52:15.255525 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.255502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.262544 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.262255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bcvzt\""
Sep 18 22:52:15.262544 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.262292    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-lz7kd\""
Sep 18 22:52:15.262544 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.262435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 22:52:15.262816 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.262592    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 22:52:15.262816 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.262713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 22:52:15.267663 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.267640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:52:15.402042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.402005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vg7q\" (UniqueName: \"kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.402042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.402044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.503440 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.503350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.503565 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.503501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vg7q\" (UniqueName: \"kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.506106 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.506068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.513314 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.513283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vg7q\" (UniqueName: \"kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.572959 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.572937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:15.847573 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.847531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:52:15.851704 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:52:15.851676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc060a76_89ab_4062_89d1_32330c8b3a81.slice/crio-f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b WatchSource:0}: Error finding container f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b: Status 404 returned error can't find the container with id f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b
Sep 18 22:52:15.988644 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.988608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" event={"ID":"fc060a76-89ab-4062-89d1-32330c8b3a81","Type":"ContainerStarted","Data":"1de96cedf194dfc20b4561957145cfb9a1839d0a19de533fb5c4b37bbc744ccc"}
Sep 18 22:52:15.988800 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.988668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" event={"ID":"fc060a76-89ab-4062-89d1-32330c8b3a81","Type":"ContainerStarted","Data":"f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b"}
Sep 18 22:52:15.988800 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:15.988706    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:16.012068 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:16.012029    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podStartSLOduration=1.01201325 podStartE2EDuration="1.01201325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:52:16.01131785 +0000 UTC m=+3676.210026151" watchObservedRunningTime="2026-09-18 22:52:16.01201325 +0000 UTC m=+3676.210721548"
Sep 18 22:52:16.876986 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:16.876951    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:52:16.877333 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:16.877127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:52:28.876562 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:28.876532    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:52:28.876934 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:28.876709    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:52:34.750647 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:34.750604    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:52:34.755884 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:34.755856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:52:34.761734 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:34.761696    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:52:34.761734 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:52:34.761734 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:52:34.761734 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035" cmd=["/bin/sleep","15"]
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:34.761763    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" message="" err=<
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:34.761782    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main" containerID="cri-o://f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035"
Sep 18 22:52:34.762054 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:34.761829    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main" containerID="cri-o://f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035" gracePeriod=60
Sep 18 22:52:35.082118 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.082036    2563 generic.go:356] "Generic (PLEG): container finished" podID="580e659b-8f6a-4a84-8668-98690183816e" containerID="f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035" exitCode=0
Sep 18 22:52:35.082249 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.082117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" event={"ID":"580e659b-8f6a-4a84-8668-98690183816e","Type":"ContainerDied","Data":"f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035"}
Sep 18 22:52:35.500443 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.500418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:52:35.547834 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.547975 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.547975 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.547975 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.547975 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw7q8\" (UniqueName: \"kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.548195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.547981    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm\") pod \"580e659b-8f6a-4a84-8668-98690183816e\" (UID: \"580e659b-8f6a-4a84-8668-98690183816e\") "
Sep 18 22:52:35.548195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.548079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:52:35.548306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.548233    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:35.548568 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.548442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home" (OuterVolumeSpecName: "home") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:52:35.548691 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.548618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache" (OuterVolumeSpecName: "model-cache") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:52:35.550927 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.550898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm" (OuterVolumeSpecName: "dshm") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:52:35.551025 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.550936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8" (OuterVolumeSpecName: "kube-api-access-fw7q8") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "kube-api-access-fw7q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:52:35.551025 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.551017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "580e659b-8f6a-4a84-8668-98690183816e" (UID: "580e659b-8f6a-4a84-8668-98690183816e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:52:35.649263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.649193    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:35.649263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.649215    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:35.649263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.649225    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580e659b-8f6a-4a84-8668-98690183816e-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:35.649263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.649233    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7q8\" (UniqueName: \"kubernetes.io/projected/580e659b-8f6a-4a84-8668-98690183816e-kube-api-access-fw7q8\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:35.649263 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:35.649241    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580e659b-8f6a-4a84-8668-98690183816e-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:52:36.087545 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.087516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" event={"ID":"580e659b-8f6a-4a84-8668-98690183816e","Type":"ContainerDied","Data":"7fdea2bfa83c2ad374ad41fad3a31460532c5e902c92f427ed50fb6250804eb1"}
Sep 18 22:52:36.087951 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.087560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"
Sep 18 22:52:36.087951 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.087574    2563 scope.go:117] "RemoveContainer" containerID="f504fc615f44577cc717f0272616800aaab2d6a7913fa1ab368f01fd42732035"
Sep 18 22:52:36.105171 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.105145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:52:36.108863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.108842    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn"]
Sep 18 22:52:36.392163 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.392073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:52:36.432923 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:36.432889    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ctvpn" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Sep 18 22:52:37.888079 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:37.888046    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580e659b-8f6a-4a84-8668-98690183816e" path="/var/lib/kubelet/pods/580e659b-8f6a-4a84-8668-98690183816e/volumes"
Sep 18 22:52:42.876969 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:42.876939    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:52:42.877332 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:42.877108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:52:46.392670 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:46.392623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:52:46.996535 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:46.996498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:52:49.777879 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:49.777843    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" containerID="cri-o://2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9" gracePeriod=45
Sep 18 22:52:56.392116 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:56.392079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:52:56.392538 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:56.392190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:52:57.876675 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:52:57.876644    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:52:57.877046 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:52:57.876816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:53:06.392583 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:06.392545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:53:09.877517 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:09.877486    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:53:09.877902 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:53:09.877642    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:53:16.391936 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:16.391901    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:53:20.027346 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.027319    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:53:20.163758 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.163732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r5mt\" (UniqueName: \"kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt\") pod \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") "
Sep 18 22:53:20.163916 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.163788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs\") pod \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\" (UID: \"64b4d540-cf78-4303-a402-ccd7e7cb49ef\") "
Sep 18 22:53:20.165823 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.165796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt" (OuterVolumeSpecName: "kube-api-access-9r5mt") pod "64b4d540-cf78-4303-a402-ccd7e7cb49ef" (UID: "64b4d540-cf78-4303-a402-ccd7e7cb49ef"). InnerVolumeSpecName "kube-api-access-9r5mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:53:20.165924 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.165817    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64b4d540-cf78-4303-a402-ccd7e7cb49ef" (UID: "64b4d540-cf78-4303-a402-ccd7e7cb49ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:53:20.264529 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.264495    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5mt\" (UniqueName: \"kubernetes.io/projected/64b4d540-cf78-4303-a402-ccd7e7cb49ef-kube-api-access-9r5mt\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:53:20.264529 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.264526    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b4d540-cf78-4303-a402-ccd7e7cb49ef-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:53:20.283705 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.283676    2563 generic.go:356] "Generic (PLEG): container finished" podID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerID="2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9" exitCode=0
Sep 18 22:53:20.283836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.283727    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"
Sep 18 22:53:20.283836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.283759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" event={"ID":"64b4d540-cf78-4303-a402-ccd7e7cb49ef","Type":"ContainerDied","Data":"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"}
Sep 18 22:53:20.283836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.283809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8" event={"ID":"64b4d540-cf78-4303-a402-ccd7e7cb49ef","Type":"ContainerDied","Data":"891f9b2047bc076042a0746d7e06cb7b6fe3372d432e5ee8765706aaba497045"}
Sep 18 22:53:20.283836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.283834    2563 scope.go:117] "RemoveContainer" containerID="2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"
Sep 18 22:53:20.292849 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.292827    2563 scope.go:117] "RemoveContainer" containerID="2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"
Sep 18 22:53:20.293133 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:53:20.293114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9\": container with ID starting with 2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9 not found: ID does not exist" containerID="2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"
Sep 18 22:53:20.293200 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.293140    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9"} err="failed to get container status \"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9\": rpc error: code = NotFound desc = could not find container \"2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9\": container with ID starting with 2d4236f810fa74d832765990f80e0303b62b4730b60585d7917200a595527ae9 not found: ID does not exist"
Sep 18 22:53:20.305306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.305284    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:53:20.311100 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:20.311080    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scherstp8"]
Sep 18 22:53:21.878744 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:21.878711    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:53:21.879178 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:53:21.878918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:53:21.882187 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:21.882160    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" path="/var/lib/kubelet/pods/64b4d540-cf78-4303-a402-ccd7e7cb49ef/volumes"
Sep 18 22:53:30.766074 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766034    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766340    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main"
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main"
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main"
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766367    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main"
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766443    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b4d540-cf78-4303-a402-ccd7e7cb49ef" containerName="main"
Sep 18 22:53:30.766463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.766455    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="580e659b-8f6a-4a84-8668-98690183816e" containerName="main"
Sep 18 22:53:30.770842 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.770818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.774769 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.774748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 22:53:30.775111 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.775092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 22:53:30.775196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.775104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-rfwtv\""
Sep 18 22:53:30.775196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.775092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-2rbzj\""
Sep 18 22:53:30.775196 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.775099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 22:53:30.782390 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.782371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:53:30.841503 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.841482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.841591 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.841520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46x8n\" (UniqueName: \"kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.942017 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.941983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.942133 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.942024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46x8n\" (UniqueName: \"kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.944373 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.944353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:30.952033 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:30.952010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46x8n\" (UniqueName: \"kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:31.082562 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:31.082474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:31.356101 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:31.356071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:53:31.359546 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:53:31.359508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5b7b9c_714a_4ce0_9c26_b7a1d9577b13.slice/crio-09e7caaf8470eca59a00a4fcc8f3b3513de435f640c831cae03a60f2a1132fb4 WatchSource:0}: Error finding container 09e7caaf8470eca59a00a4fcc8f3b3513de435f640c831cae03a60f2a1132fb4: Status 404 returned error can't find the container with id 09e7caaf8470eca59a00a4fcc8f3b3513de435f640c831cae03a60f2a1132fb4
Sep 18 22:53:32.338131 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:32.338088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" event={"ID":"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13","Type":"ContainerStarted","Data":"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"}
Sep 18 22:53:32.338589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:32.338158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" event={"ID":"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13","Type":"ContainerStarted","Data":"09e7caaf8470eca59a00a4fcc8f3b3513de435f640c831cae03a60f2a1132fb4"}
Sep 18 22:53:32.338589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:32.338203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:53:32.363551 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:32.363497    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podStartSLOduration=2.363480623 podStartE2EDuration="2.363480623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:53:32.360599846 +0000 UTC m=+3752.559308144" watchObservedRunningTime="2026-09-18 22:53:32.363480623 +0000 UTC m=+3752.562189247"
Sep 18 22:53:32.876562 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:32.876531    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:53:32.876741 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:53:32.876722    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:53:45.882609 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:53:45.882538    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:53:45.883065 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:53:45.882715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:54:00.876486 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:00.876449    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:54:00.876895 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:54:00.876651    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:54:03.344697 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:03.344668    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:54:15.877129 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:15.877097    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:54:15.877632 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:54:15.877331    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:54:24.537486 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:24.537451    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:54:26.996385 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:26.996347    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:54:29.877039 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:29.877003    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:54:29.877493 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:54:29.877206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:54:36.996341 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:36.996300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:54:39.559476 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:39.559432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" containerID="cri-o://1de96cedf194dfc20b4561957145cfb9a1839d0a19de533fb5c4b37bbc744ccc" gracePeriod=45
Sep 18 22:54:42.876613 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:42.876581    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:54:42.877056 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:54:42.876785    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:54:46.996455 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:46.996414    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:54:46.996855 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:46.996544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:54:56.995913 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:56.995873    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:54:57.877359 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:54:57.877324    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:54:57.877580 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:54:57.877552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:06.995879 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:06.995835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:55:09.787996 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.787968    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerID="1de96cedf194dfc20b4561957145cfb9a1839d0a19de533fb5c4b37bbc744ccc" exitCode=0
Sep 18 22:55:09.788306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.788040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" event={"ID":"fc060a76-89ab-4062-89d1-32330c8b3a81","Type":"ContainerDied","Data":"1de96cedf194dfc20b4561957145cfb9a1839d0a19de533fb5c4b37bbc744ccc"}
Sep 18 22:55:09.788306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.788091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l" event={"ID":"fc060a76-89ab-4062-89d1-32330c8b3a81","Type":"ContainerDied","Data":"f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b"}
Sep 18 22:55:09.788306 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.788113    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f483a2bab2536e8f40eb82fb5273c8ed8f1b61c8b8d020330a37c2f1f913e31b"
Sep 18 22:55:09.797542 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.797523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:55:09.882856 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.882837    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:55:09.883025 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:09.883007    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:09.905117 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.905086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vg7q\" (UniqueName: \"kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q\") pod \"fc060a76-89ab-4062-89d1-32330c8b3a81\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") "
Sep 18 22:55:09.905226 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.905163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs\") pod \"fc060a76-89ab-4062-89d1-32330c8b3a81\" (UID: \"fc060a76-89ab-4062-89d1-32330c8b3a81\") "
Sep 18 22:55:09.907243 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.907219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q" (OuterVolumeSpecName: "kube-api-access-4vg7q") pod "fc060a76-89ab-4062-89d1-32330c8b3a81" (UID: "fc060a76-89ab-4062-89d1-32330c8b3a81"). InnerVolumeSpecName "kube-api-access-4vg7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:55:09.907335 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:09.907307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc060a76-89ab-4062-89d1-32330c8b3a81" (UID: "fc060a76-89ab-4062-89d1-32330c8b3a81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:55:10.005765 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:10.005741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vg7q\" (UniqueName: \"kubernetes.io/projected/fc060a76-89ab-4062-89d1-32330c8b3a81-kube-api-access-4vg7q\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:55:10.005878 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:10.005764    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc060a76-89ab-4062-89d1-32330c8b3a81-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:55:10.792334 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:10.792296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"
Sep 18 22:55:10.813195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:10.813163    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:55:10.816106 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:10.816082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-7fb7dghl2l"]
Sep 18 22:55:11.881754 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:11.881717    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" path="/var/lib/kubelet/pods/fc060a76-89ab-4062-89d1-32330c8b3a81/volumes"
Sep 18 22:55:14.948897 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.948816    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:55:14.949325 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.949128    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main"
Sep 18 22:55:14.949325 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.949140    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main"
Sep 18 22:55:14.949325 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.949216    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc060a76-89ab-4062-89d1-32330c8b3a81" containerName="main"
Sep 18 22:55:14.952181 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.952157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:14.954621 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.954595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:55:14.954863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.954844    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 22:55:14.954973 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.954877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-c6jn8\""
Sep 18 22:55:14.954973 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.954850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 22:55:14.964757 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:14.964730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:55:15.040830 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.040941 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.040941 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.040941 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.041069 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.041069 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.040985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.041069 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.041003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.041069 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.041020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.041069 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.041036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdn4j\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.141901 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.141982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdn4j\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142040 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142458 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142458 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142552 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142552 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.142840 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.142821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.144552 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.144523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.144662 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.144570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.150945 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.150916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.151240 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.151201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdn4j\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j\") pod \"router-gateway-2-openshift-default-5b4cfc9d8-tv2bf\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.263970 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.263908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:15.541495 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.541419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:55:15.545066 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:55:15.545020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be6de1f_c3bb_4cc6_91f5_e252ffc39d37.slice/crio-c6267699b7a840f475bf2c7907de12798835211e7a6fbb0fedab55a9f7c549be WatchSource:0}: Error finding container c6267699b7a840f475bf2c7907de12798835211e7a6fbb0fedab55a9f7c549be: Status 404 returned error can't find the container with id c6267699b7a840f475bf2c7907de12798835211e7a6fbb0fedab55a9f7c549be
Sep 18 22:55:15.547742 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.547688    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:15.547873 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.547773    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:15.547873 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.547820    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:15.824802 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.824710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerStarted","Data":"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"}
Sep 18 22:55:15.824967 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.824806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerStarted","Data":"c6267699b7a840f475bf2c7907de12798835211e7a6fbb0fedab55a9f7c549be"}
Sep 18 22:55:15.853789 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:15.853732    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podStartSLOduration=1.853714047 podStartE2EDuration="1.853714047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:55:15.850868113 +0000 UTC m=+3856.049576412" watchObservedRunningTime="2026-09-18 22:55:15.853714047 +0000 UTC m=+3856.052422348"
Sep 18 22:55:16.264786 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:16.264751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:16.269656 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:16.269630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:16.829472 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:16.829436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:16.830328 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:16.830305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:19.738951 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:19.738918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:55:22.876366 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:22.876335    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:55:23.348619 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.348584    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:55:23.873324 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.873297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/14.log"
Sep 18 22:55:23.873841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.873825    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/13.log"
Sep 18 22:55:23.873931 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.873861    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6" exitCode=1
Sep 18 22:55:23.873987 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.873949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"}
Sep 18 22:55:23.874049 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.874028    2563 scope.go:117] "RemoveContainer" containerID="738499a1612115c4927908ef30b5dc9d3dd98cead2408cdd653ff3fac430d4cb"
Sep 18 22:55:23.874435 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:23.874417    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:55:23.874659 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:23.874639    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:24.301996 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:24.301911    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:55:24.879900 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:24.879875    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/14.log"
Sep 18 22:55:24.880230 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:24.880215    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:55:24.880494 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:24.880461    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:24.964048 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:24.964014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 22:55:25.884694 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:25.884666    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:55:25.885067 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:25.884854    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:29.903589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.903557    2563 generic.go:356] "Generic (PLEG): container finished" podID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerID="7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1" exitCode=0
Sep 18 22:55:29.903589 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.903595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerDied","Data":"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"}
Sep 18 22:55:29.904183 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.904038    2563 scope.go:117] "RemoveContainer" containerID="7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"
Sep 18 22:55:29.904980 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.904949    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:29.905078 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.905036    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:29.905140 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:29.905084    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:55:30.909645 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:30.909613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerStarted","Data":"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"}
Sep 18 22:55:31.265135 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:31.265031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:31.266670 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:31.266636    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 18 22:55:31.830023 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:31.829990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:32.265718 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:32.265275    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 18 22:55:33.264935 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:33.264891    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 18 22:55:33.348580 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:33.348543    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:55:34.265002 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:34.264950    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 18 22:55:34.768247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:34.768192    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" containerID="cri-o://75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f" gracePeriod=45
Sep 18 22:55:35.264853 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:35.264801    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 18 22:55:36.268662 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:36.268630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:36.269691 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:36.269670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:55:40.876702 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:40.876674    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:55:40.877090 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:40.876838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:55:43.344971 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:43.344934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:55:43.345469 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:43.345036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:55:49.370453 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.370322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:55:49.372958 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.372934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.375866 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.375844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6mhsb\""
Sep 18 22:55:49.376149 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.376129    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 22:55:49.376668 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.376644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-6brpc\""
Sep 18 22:55:49.383107 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.383083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:55:49.500882 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.500855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbtr\" (UniqueName: \"kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.501034 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.500893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.602198 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.602168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbtr\" (UniqueName: \"kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.602335 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.602204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.604630 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.604610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.610014 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.609988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbtr\" (UniqueName: \"kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr\") pod \"router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.686584 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.686508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:49.958039 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.958013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:55:49.960984 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:55:49.960956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f515c6_ef6c_4baa_89f0_dcddca7a3cbc.slice/crio-1b8a489feae2491e9c52c9482d942d3fba1651e500db4d3247692a93839a0484 WatchSource:0}: Error finding container 1b8a489feae2491e9c52c9482d942d3fba1651e500db4d3247692a93839a0484: Status 404 returned error can't find the container with id 1b8a489feae2491e9c52c9482d942d3fba1651e500db4d3247692a93839a0484
Sep 18 22:55:49.991502 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:49.991478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" event={"ID":"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc","Type":"ContainerStarted","Data":"1b8a489feae2491e9c52c9482d942d3fba1651e500db4d3247692a93839a0484"}
Sep 18 22:55:50.999026 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:50.998992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" event={"ID":"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc","Type":"ContainerStarted","Data":"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"}
Sep 18 22:55:50.999413 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:50.999056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:55:51.016337 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:51.016289    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podStartSLOduration=2.016275635 podStartE2EDuration="2.016275635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:55:51.015301233 +0000 UTC m=+3891.214009525" watchObservedRunningTime="2026-09-18 22:55:51.016275635 +0000 UTC m=+3891.214983934"
Sep 18 22:55:53.344289 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:53.344255    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:55:53.877057 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:55:53.877030    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:55:53.877263 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:55:53.877190    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:56:02.141018 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:02.140988    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/14.log"
Sep 18 22:56:02.153777 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:02.153750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:56:02.155911 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:02.155879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/14.log"
Sep 18 22:56:02.167530 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:02.167512    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 22:56:03.344620 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:03.344584    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:56:05.017547 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.017522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:56:05.062908 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.062837    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerID="75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f" exitCode=0
Sep 18 22:56:05.062908 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.062884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"
Sep 18 22:56:05.062908 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.062894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" event={"ID":"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13","Type":"ContainerDied","Data":"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"}
Sep 18 22:56:05.063085 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.062955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb" event={"ID":"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13","Type":"ContainerDied","Data":"09e7caaf8470eca59a00a4fcc8f3b3513de435f640c831cae03a60f2a1132fb4"}
Sep 18 22:56:05.063085 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.062982    2563 scope.go:117] "RemoveContainer" containerID="75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"
Sep 18 22:56:05.071041 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.071024    2563 scope.go:117] "RemoveContainer" containerID="75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"
Sep 18 22:56:05.071289 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:05.071271    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f\": container with ID starting with 75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f not found: ID does not exist" containerID="75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"
Sep 18 22:56:05.071351 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.071299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f"} err="failed to get container status \"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f\": rpc error: code = NotFound desc = could not find container \"75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f\": container with ID starting with 75a638da74960d8d723f9c23a25a09cbd2503fab483f8aab16dea7e73153d06f not found: ID does not exist"
Sep 18 22:56:05.126055 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.126021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs\") pod \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") "
Sep 18 22:56:05.126166 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.126086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46x8n\" (UniqueName: \"kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n\") pod \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\" (UID: \"cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13\") "
Sep 18 22:56:05.128166 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.128138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n" (OuterVolumeSpecName: "kube-api-access-46x8n") pod "cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" (UID: "cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13"). InnerVolumeSpecName "kube-api-access-46x8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:56:05.128249 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.128171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" (UID: "cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:56:05.227550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.227531    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:56:05.227550 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.227550    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46x8n\" (UniqueName: \"kubernetes.io/projected/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13-kube-api-access-46x8n\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:56:05.385139 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.385108    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:56:05.392220 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.392194    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelzjwb"]
Sep 18 22:56:05.876700 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.876672    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:56:05.876880 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:05.876834    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:56:05.882867 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:05.882842    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" path="/var/lib/kubelet/pods/cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13/volumes"
Sep 18 22:56:14.689104 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.689073    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:56:14.689477 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.689380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main"
Sep 18 22:56:14.689477 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.689415    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main"
Sep 18 22:56:14.689477 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.689478    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5b7b9c-714a-4ce0-9c26-b7a1d9577b13" containerName="main"
Sep 18 22:56:14.692327 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.692311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.707966 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.707944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-4dm7n\""
Sep 18 22:56:14.708082 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.707944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-25m6l\""
Sep 18 22:56:14.708211 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.708197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 22:56:14.715235 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.715221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 22:56:14.720252 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.720233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 22:56:14.729697 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.729675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:56:14.799743 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.799712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxbk\" (UniqueName: \"kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.799860 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.799763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.900360 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.900331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.900480 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.900388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxbk\" (UniqueName: \"kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.902826 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.902810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:14.908707 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:14.908684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxbk\" (UniqueName: \"kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:15.006618 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:15.006566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:15.284635 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:15.284527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:56:15.287305 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:56:15.287275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5409e0_c96e_4380_b66f_01866d902d04.slice/crio-a65b938078036d539b968cc6268ad498823b001439c6b0262f858137ae624f76 WatchSource:0}: Error finding container a65b938078036d539b968cc6268ad498823b001439c6b0262f858137ae624f76: Status 404 returned error can't find the container with id a65b938078036d539b968cc6268ad498823b001439c6b0262f858137ae624f76
Sep 18 22:56:16.114824 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:16.114783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" event={"ID":"df5409e0-c96e-4380-b66f-01866d902d04","Type":"ContainerStarted","Data":"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"}
Sep 18 22:56:16.115217 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:16.114840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" event={"ID":"df5409e0-c96e-4380-b66f-01866d902d04","Type":"ContainerStarted","Data":"a65b938078036d539b968cc6268ad498823b001439c6b0262f858137ae624f76"}
Sep 18 22:56:16.115217 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:16.114877    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:16.146836 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:16.146787    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podStartSLOduration=2.146773039 podStartE2EDuration="2.146773039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:56:16.1426721 +0000 UTC m=+3916.341380399" watchObservedRunningTime="2026-09-18 22:56:16.146773039 +0000 UTC m=+3916.345481338"
Sep 18 22:56:19.881793 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:19.881765    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:56:19.882151 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:19.881922    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:56:22.005078 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:22.005047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:56:30.876409 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:30.876369    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:56:30.876889 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:30.876577    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:56:45.876815 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:45.876735    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:56:45.877270 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:45.876968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:56:47.121688 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:47.121662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:56:59.882137 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:56:59.882108    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:56:59.882537 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:56:59.882262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:57:14.880111 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:14.880076    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:57:14.880626 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:57:14.880300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:57:29.877215 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:29.877180    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:57:29.877862 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:57:29.877358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:57:36.340446 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:36.340409    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:57:37.121674 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:37.121636    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:57:40.876500 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:40.876466    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:57:40.876892 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:57:40.876645    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:57:47.121949 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:47.121905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:57:51.363027 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:51.362961    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" containerID="cri-o://08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d" gracePeriod=45
Sep 18 22:57:53.877042 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:53.877010    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:57:53.877446 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:57:53.877205    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:57:57.121532 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:57.121490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:57:57.122028 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:57.121644    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:57:57.859164 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:57:57.859123    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:58:02.004498 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:02.004467    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:04.877201 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:04.877169    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:58:04.877615 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:04.877357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:58:07.121624 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:07.121588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:12.004231 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:12.004196    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:12.878881 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:12.878847    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" containerID="cri-o://cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3" gracePeriod=45
Sep 18 22:58:17.121753 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:17.121674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:18.876891 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:18.876856    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:58:18.877315 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:18.877032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:58:21.609049 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.609020    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:58:21.654628 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.654558    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxbk\" (UniqueName: \"kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk\") pod \"df5409e0-c96e-4380-b66f-01866d902d04\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") "
Sep 18 22:58:21.654778 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.654652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs\") pod \"df5409e0-c96e-4380-b66f-01866d902d04\" (UID: \"df5409e0-c96e-4380-b66f-01866d902d04\") "
Sep 18 22:58:21.656749 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.656713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df5409e0-c96e-4380-b66f-01866d902d04" (UID: "df5409e0-c96e-4380-b66f-01866d902d04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:58:21.656878 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.656766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk" (OuterVolumeSpecName: "kube-api-access-7bxbk") pod "df5409e0-c96e-4380-b66f-01866d902d04" (UID: "df5409e0-c96e-4380-b66f-01866d902d04"). InnerVolumeSpecName "kube-api-access-7bxbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:21.701820 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.701791    2563 generic.go:356] "Generic (PLEG): container finished" podID="df5409e0-c96e-4380-b66f-01866d902d04" containerID="08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d" exitCode=0
Sep 18 22:58:21.701935 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.701834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"
Sep 18 22:58:21.701935 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.701871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" event={"ID":"df5409e0-c96e-4380-b66f-01866d902d04","Type":"ContainerDied","Data":"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"}
Sep 18 22:58:21.701935 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.701918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h" event={"ID":"df5409e0-c96e-4380-b66f-01866d902d04","Type":"ContainerDied","Data":"a65b938078036d539b968cc6268ad498823b001439c6b0262f858137ae624f76"}
Sep 18 22:58:21.702051 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.701940    2563 scope.go:117] "RemoveContainer" containerID="08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"
Sep 18 22:58:21.710572 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.710556    2563 scope.go:117] "RemoveContainer" containerID="08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"
Sep 18 22:58:21.710831 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:21.710813    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d\": container with ID starting with 08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d not found: ID does not exist" containerID="08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"
Sep 18 22:58:21.710897 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.710836    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d"} err="failed to get container status \"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d\": rpc error: code = NotFound desc = could not find container \"08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d\": container with ID starting with 08c1ce8845042219ed86d8c8c17e398b34ee922659364b5246827233e917c24d not found: ID does not exist"
Sep 18 22:58:21.723029 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.723004    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:58:21.728364 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.728343    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepvz8h"]
Sep 18 22:58:21.755372 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.755350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxbk\" (UniqueName: \"kubernetes.io/projected/df5409e0-c96e-4380-b66f-01866d902d04-kube-api-access-7bxbk\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:21.755463 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.755370    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5409e0-c96e-4380-b66f-01866d902d04-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:21.881637 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:21.881617    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5409e0-c96e-4380-b66f-01866d902d04" path="/var/lib/kubelet/pods/df5409e0-c96e-4380-b66f-01866d902d04/volumes"
Sep 18 22:58:22.005275 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:22.005209    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:22.005377 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:22.005300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:58:32.004453 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:32.004413    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:32.876570 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:32.876545    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:58:32.876747 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:32.876712    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:58:34.120841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.120803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 22:58:34.121316 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.121267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main"
Sep 18 22:58:34.121316 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.121285    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main"
Sep 18 22:58:34.121468 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.121408    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5409e0-c96e-4380-b66f-01866d902d04" containerName="main"
Sep 18 22:58:34.125956 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.125933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.128867 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.128838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-fp2ct\""
Sep 18 22:58:34.128997 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.128877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-4kc2m\""
Sep 18 22:58:34.128997 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.128845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 22:58:34.128997 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.128852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 22:58:34.129266 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.129117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 22:58:34.132109 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.132085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 22:58:34.252625 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.252596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5lc\" (UniqueName: \"kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.252765 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.252660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.354018 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.353985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5lc\" (UniqueName: \"kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.354155 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.354033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.356612 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.356594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.362147 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.362129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5lc\" (UniqueName: \"kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.438279 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.438219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:34.718907 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.718823    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 22:58:34.722276 ip-10-0-141-13 kubenswrapper[2563]: W0918 22:58:34.722251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bedd565_25b8_4a45_a744_cf352b713eb2.slice/crio-f9d6dfcdf2d611ae3ebbe0ff26b7133906577380d8f921cea4d92ec97f6f4920 WatchSource:0}: Error finding container f9d6dfcdf2d611ae3ebbe0ff26b7133906577380d8f921cea4d92ec97f6f4920: Status 404 returned error can't find the container with id f9d6dfcdf2d611ae3ebbe0ff26b7133906577380d8f921cea4d92ec97f6f4920
Sep 18 22:58:34.762917 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:34.762884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" event={"ID":"5bedd565-25b8-4a45-a744-cf352b713eb2","Type":"ContainerStarted","Data":"f9d6dfcdf2d611ae3ebbe0ff26b7133906577380d8f921cea4d92ec97f6f4920"}
Sep 18 22:58:35.769378 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:35.769341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" event={"ID":"5bedd565-25b8-4a45-a744-cf352b713eb2","Type":"ContainerStarted","Data":"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"}
Sep 18 22:58:35.769841 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:35.769446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:58:35.787062 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:35.787017    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podStartSLOduration=1.7870028470000001 podStartE2EDuration="1.787002847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:58:35.785868452 +0000 UTC m=+4055.984576765" watchObservedRunningTime="2026-09-18 22:58:35.787002847 +0000 UTC m=+4055.985711160"
Sep 18 22:58:42.004696 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:42.004660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:58:43.125077 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.125053    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:58:43.229333 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.229269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs\") pod \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") "
Sep 18 22:58:43.229333 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.229308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndbtr\" (UniqueName: \"kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr\") pod \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\" (UID: \"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc\") "
Sep 18 22:58:43.231513 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.231480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" (UID: "89f515c6-ef6c-4baa-89f0-dcddca7a3cbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:58:43.231614 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.231553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr" (OuterVolumeSpecName: "kube-api-access-ndbtr") pod "89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" (UID: "89f515c6-ef6c-4baa-89f0-dcddca7a3cbc"). InnerVolumeSpecName "kube-api-access-ndbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:43.335222 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.330166    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:43.335222 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.330247    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbtr\" (UniqueName: \"kubernetes.io/projected/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc-kube-api-access-ndbtr\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:43.806715 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.806681    2563 generic.go:356] "Generic (PLEG): container finished" podID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerID="cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3" exitCode=0
Sep 18 22:58:43.806863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.806732    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"
Sep 18 22:58:43.806863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.806770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" event={"ID":"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc","Type":"ContainerDied","Data":"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"}
Sep 18 22:58:43.806863 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.806847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7" event={"ID":"89f515c6-ef6c-4baa-89f0-dcddca7a3cbc","Type":"ContainerDied","Data":"1b8a489feae2491e9c52c9482d942d3fba1651e500db4d3247692a93839a0484"}
Sep 18 22:58:43.806988 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.806869    2563 scope.go:117] "RemoveContainer" containerID="cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"
Sep 18 22:58:43.815267 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.815249    2563 scope.go:117] "RemoveContainer" containerID="cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"
Sep 18 22:58:43.815563 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:43.815542    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3\": container with ID starting with cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3 not found: ID does not exist" containerID="cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"
Sep 18 22:58:43.815638 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.815570    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3"} err="failed to get container status \"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3\": rpc error: code = NotFound desc = could not find container \"cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3\": container with ID starting with cba99843386452a5ed119b7de864afb184a5c7e9707389ed52be23d0c16f15e3 not found: ID does not exist"
Sep 18 22:58:43.827256 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.827235    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:58:43.831231 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.831206    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-6b774cb6-bwgf7"]
Sep 18 22:58:43.881491 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:43.881473    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" path="/var/lib/kubelet/pods/89f515c6-ef6c-4baa-89f0-dcddca7a3cbc/volumes"
Sep 18 22:58:46.876995 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:46.876967    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:58:46.877359 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:46.877144    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:58:52.928672 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:52.928639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:58:52.929110 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:52.928816    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy" containerID="cri-o://c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488" gracePeriod=30
Sep 18 22:58:58.183775 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.183751    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:58:58.344494 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344621 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344621 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344621 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdn4j\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.344791 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy\") pod \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\" (UID: \"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37\") "
Sep 18 22:58:58.345037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:58:58.345037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.344885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:58.345037 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.345027    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istiod-ca-cert\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.345195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.345047    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-socket\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.345195 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.345128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:58.345295 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.345213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data" (OuterVolumeSpecName: "istio-data") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:58.345295 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.345233    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:58.346727 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.346706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token" (OuterVolumeSpecName: "istio-token") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:58.347166 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.347145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:58.347247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.347180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:58:58.347247 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.347184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j" (OuterVolumeSpecName: "kube-api-access-fdn4j") pod "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" (UID: "2be6de1f-c3bb-4cc6-91f5-e252ffc39d37"). InnerVolumeSpecName "kube-api-access-fdn4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:58.445763 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445738    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-data\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445765    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-credential-socket\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445781    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdn4j\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-kube-api-access-fdn4j\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445794    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-workload-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445832    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-envoy\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445845    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-token\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.445859 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.445858    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37-istio-podinfo\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 22:58:58.878185 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.878145    2563 generic.go:356] "Generic (PLEG): container finished" podID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerID="c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488" exitCode=0
Sep 18 22:58:58.878359 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.878203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"
Sep 18 22:58:58.878359 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.878206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerDied","Data":"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"}
Sep 18 22:58:58.878359 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.878338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf" event={"ID":"2be6de1f-c3bb-4cc6-91f5-e252ffc39d37","Type":"ContainerDied","Data":"c6267699b7a840f475bf2c7907de12798835211e7a6fbb0fedab55a9f7c549be"}
Sep 18 22:58:58.878537 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.878373    2563 scope.go:117] "RemoveContainer" containerID="c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"
Sep 18 22:58:58.887473 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.887456    2563 scope.go:117] "RemoveContainer" containerID="7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"
Sep 18 22:58:58.894939 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.894925    2563 scope.go:117] "RemoveContainer" containerID="c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"
Sep 18 22:58:58.895194 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:58.895174    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488\": container with ID starting with c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488 not found: ID does not exist" containerID="c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"
Sep 18 22:58:58.895238 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.895201    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488"} err="failed to get container status \"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488\": rpc error: code = NotFound desc = could not find container \"c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488\": container with ID starting with c87a7e03d1dbfa26c1b04bb9f2505b2d8aec25ae5e8399ea798964032f485488 not found: ID does not exist"
Sep 18 22:58:58.895238 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.895219    2563 scope.go:117] "RemoveContainer" containerID="7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"
Sep 18 22:58:58.895470 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:58.895449    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1\": container with ID starting with 7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1 not found: ID does not exist" containerID="7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"
Sep 18 22:58:58.895527 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.895479    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1"} err="failed to get container status \"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1\": rpc error: code = NotFound desc = could not find container \"7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1\": container with ID starting with 7f83c23421daada92b24cfa023f302a208e9c6382e04ad135346d2180aa4cdf1 not found: ID does not exist"
Sep 18 22:58:58.908175 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.908143    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:58:58.917034 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:58.917013    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5b4cfc9d8-tv2bf"]
Sep 18 22:58:59.877481 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:59.877449    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:58:59.877987 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:58:59.877726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:58:59.883085 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:58:59.883063    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" path="/var/lib/kubelet/pods/2be6de1f-c3bb-4cc6-91f5-e252ffc39d37/volumes"
Sep 18 22:59:06.775254 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:06.775223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 22:59:08.708862 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:08.708830    2563 scope.go:117] "RemoveContainer" containerID="1de96cedf194dfc20b4561957145cfb9a1839d0a19de533fb5c4b37bbc744ccc"
Sep 18 22:59:11.879498 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:11.879469    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:59:11.879872 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:59:11.879642    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:59:23.877022 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:23.876986    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:59:23.877533 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:59:23.877160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:59:36.876913 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:36.876880    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:59:36.877372 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:59:36.877025    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:59:47.877140 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:47.877065    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:59:47.877586 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:59:47.877229    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 22:59:58.876946 ip-10-0-141-13 kubenswrapper[2563]: I0918 22:59:58.876915    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 22:59:58.877325 ip-10-0-141-13 kubenswrapper[2563]: E0918 22:59:58.877089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:12.876293 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:12.876266    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 23:00:12.876729 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:00:12.876462    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:25.876665 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:25.876633    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 23:00:26.249625 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.249543    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/15.log"
Sep 18 23:00:26.250017 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.250001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/14.log"
Sep 18 23:00:26.250066 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.250036    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6" exitCode=1
Sep 18 23:00:26.250109 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.250095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"}
Sep 18 23:00:26.250153 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.250144    2563 scope.go:117] "RemoveContainer" containerID="904e79fd3df1b898d3f0c9b7a0bbac8e7d20e9e33817c8d10e41c133607357f6"
Sep 18 23:00:26.250496 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:26.250468    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:00:26.250668 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:00:26.250648    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:27.256103 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:27.256072    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/15.log"
Sep 18 23:00:34.302213 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:34.302178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:00:34.304727 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:34.302554    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:00:34.304727 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:00:34.302713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:34.964174 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:34.964144    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:00:35.291742 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:35.291657    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:00:35.291921 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:00:35.291892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:35.959917 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:35.959876    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 23:00:36.776131 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:36.776093    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:00:46.775233 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:46.775198    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:00:47.876625 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:47.876589    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:00:47.877018 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:00:47.876780    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:00:50.982051 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:50.982008    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" containerID="cri-o://84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f" gracePeriod=45
Sep 18 23:00:56.247437 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247781    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247811    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247819    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247835    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main"
Sep 18 23:00:56.247901 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main"
Sep 18 23:00:56.248251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247914    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.248251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247923    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f515c6-ef6c-4baa-89f0-dcddca7a3cbc" containerName="main"
Sep 18 23:00:56.248251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.247937    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be6de1f-c3bb-4cc6-91f5-e252ffc39d37" containerName="istio-proxy"
Sep 18 23:00:56.251127 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.251102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.253422 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.253381    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-k99mn\""
Sep 18 23:00:56.261263 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.261241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:00:56.312588 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.312557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.312588 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.312588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clww\" (UniqueName: \"kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.413937 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.413905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.414093 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.413947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clww\" (UniqueName: \"kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.416523 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.416492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.424817 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.424792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clww\" (UniqueName: \"kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww\") pod \"stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.562275 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.562214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:56.775312 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.775272    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:00:56.775492 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.775439    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 23:00:56.834337 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:56.834314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:00:56.836659 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:00:56.836634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d12a76d_e366_400b_a54d_81bad65b324a.slice/crio-99d7c3e3f51ee1bc9f11680c33b766e19ede46f02dfaa1910d177219272a876a WatchSource:0}: Error finding container 99d7c3e3f51ee1bc9f11680c33b766e19ede46f02dfaa1910d177219272a876a: Status 404 returned error can't find the container with id 99d7c3e3f51ee1bc9f11680c33b766e19ede46f02dfaa1910d177219272a876a
Sep 18 23:00:57.388450 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:57.388413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" event={"ID":"9d12a76d-e366-400b-a54d-81bad65b324a","Type":"ContainerStarted","Data":"49a516ee319505e0d48dc7e857ddb517ecc7c6748c2a6c1a2564e1d7a3f7226a"}
Sep 18 23:00:57.388952 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:57.388470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" event={"ID":"9d12a76d-e366-400b-a54d-81bad65b324a","Type":"ContainerStarted","Data":"99d7c3e3f51ee1bc9f11680c33b766e19ede46f02dfaa1910d177219272a876a"}
Sep 18 23:00:57.388952 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:57.388527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:00:57.405811 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:00:57.405771    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podStartSLOduration=1.405758182 podStartE2EDuration="1.405758182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:00:57.404896533 +0000 UTC m=+4197.603604822" watchObservedRunningTime="2026-09-18 23:00:57.405758182 +0000 UTC m=+4197.604466527"
Sep 18 23:01:00.876735 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:00.876705    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:01:00.877105 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:00.876862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:01:02.168202 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:02.168170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/15.log"
Sep 18 23:01:02.179110 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:02.179085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:01:02.182701 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:02.182682    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/15.log"
Sep 18 23:01:02.195753 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:02.195737    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:01:06.775626 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:06.775588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:01:14.877322 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:14.877247    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:01:14.877699 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:14.877433    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:01:16.775459 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:16.775425    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:01:21.229515 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.229490    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 23:01:21.321730 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.321673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl5lc\" (UniqueName: \"kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc\") pod \"5bedd565-25b8-4a45-a744-cf352b713eb2\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") "
Sep 18 23:01:21.321730 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.321706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs\") pod \"5bedd565-25b8-4a45-a744-cf352b713eb2\" (UID: \"5bedd565-25b8-4a45-a744-cf352b713eb2\") "
Sep 18 23:01:21.323904 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.323871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc" (OuterVolumeSpecName: "kube-api-access-sl5lc") pod "5bedd565-25b8-4a45-a744-cf352b713eb2" (UID: "5bedd565-25b8-4a45-a744-cf352b713eb2"). InnerVolumeSpecName "kube-api-access-sl5lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:01:21.324011 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.323958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bedd565-25b8-4a45-a744-cf352b713eb2" (UID: "5bedd565-25b8-4a45-a744-cf352b713eb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:01:21.422439 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.422414    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5lc\" (UniqueName: \"kubernetes.io/projected/5bedd565-25b8-4a45-a744-cf352b713eb2-kube-api-access-sl5lc\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:01:21.422439 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.422437    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd565-25b8-4a45-a744-cf352b713eb2-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:01:21.495849 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.495821    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerID="84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f" exitCode=0
Sep 18 23:01:21.495963 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.495860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" event={"ID":"5bedd565-25b8-4a45-a744-cf352b713eb2","Type":"ContainerDied","Data":"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"}
Sep 18 23:01:21.495963 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.495868    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"
Sep 18 23:01:21.495963 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.495905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk" event={"ID":"5bedd565-25b8-4a45-a744-cf352b713eb2","Type":"ContainerDied","Data":"f9d6dfcdf2d611ae3ebbe0ff26b7133906577380d8f921cea4d92ec97f6f4920"}
Sep 18 23:01:21.495963 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.495928    2563 scope.go:117] "RemoveContainer" containerID="84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"
Sep 18 23:01:21.504759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.504740    2563 scope.go:117] "RemoveContainer" containerID="84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"
Sep 18 23:01:21.505024 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:21.505005    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f\": container with ID starting with 84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f not found: ID does not exist" containerID="84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"
Sep 18 23:01:21.505096 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.505036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f"} err="failed to get container status \"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f\": rpc error: code = NotFound desc = could not find container \"84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f\": container with ID starting with 84f8b3d2219bfafc95f8eeef7125718256663260d6a6fd0adc66b843674b383f not found: ID does not exist"
Sep 18 23:01:21.516568 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.516542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 23:01:21.520379 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.520358    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-xrcnk"]
Sep 18 23:01:21.881687 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:21.881656    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" path="/var/lib/kubelet/pods/5bedd565-25b8-4a45-a744-cf352b713eb2/volumes"
Sep 18 23:01:28.394574 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:28.394543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:01:28.877377 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:28.877351    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:01:28.877570 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:28.877548    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:01:43.877278 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:43.877243    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:01:43.877666 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:43.877467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:01:45.073758 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.073721    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:01:45.074227 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.074193    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main"
Sep 18 23:01:45.074227 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.074210    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main"
Sep 18 23:01:45.074319 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.074298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bedd565-25b8-4a45-a744-cf352b713eb2" containerName="main"
Sep 18 23:01:45.077460 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.077439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.080716 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.080688    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 23:01:45.080837 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.080815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 23:01:45.081584 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.081557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-bjm8j\""
Sep 18 23:01:45.081707 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.081690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 23:01:45.081791 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.081772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-tm76g\""
Sep 18 23:01:45.089053 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.089033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:01:45.097354 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.097334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.097462 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.097420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425lm\" (UniqueName: \"kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.198533 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.198501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.198737 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.198630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425lm\" (UniqueName: \"kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.201251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.201228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.206876 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.206850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-425lm\" (UniqueName: \"kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.390137 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.390108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:45.661063 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:45.661036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:01:45.662625 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:01:45.662599    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177ecbad_a069_4e9c_a463_ec2a1e10d1db.slice/crio-e47d7614f43a93dbb9a59e89eb42a87075eaf37c973f7bd0c27081bb33b59c71 WatchSource:0}: Error finding container e47d7614f43a93dbb9a59e89eb42a87075eaf37c973f7bd0c27081bb33b59c71: Status 404 returned error can't find the container with id e47d7614f43a93dbb9a59e89eb42a87075eaf37c973f7bd0c27081bb33b59c71
Sep 18 23:01:46.616541 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:46.616494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" event={"ID":"177ecbad-a069-4e9c-a463-ec2a1e10d1db","Type":"ContainerStarted","Data":"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"}
Sep 18 23:01:46.617015 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:46.616559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" event={"ID":"177ecbad-a069-4e9c-a463-ec2a1e10d1db","Type":"ContainerStarted","Data":"e47d7614f43a93dbb9a59e89eb42a87075eaf37c973f7bd0c27081bb33b59c71"}
Sep 18 23:01:46.617015 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:46.616629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:01:46.652320 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:46.652268    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podStartSLOduration=1.652250327 podStartE2EDuration="1.652250327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:01:46.650778744 +0000 UTC m=+4246.849487044" watchObservedRunningTime="2026-09-18 23:01:46.652250327 +0000 UTC m=+4246.850958627"
Sep 18 23:01:55.877082 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:01:55.877045    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:01:55.877611 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:01:55.877245    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:02:06.876502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:06.876476    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:02:06.876919 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:02:06.876644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:02:17.623291 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:17.623258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:02:21.878721 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:21.878691    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:02:21.879226 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:02:21.878889    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:02:33.877373 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:33.877338    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:02:33.877854 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:02:33.877553    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:02:44.877096 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:44.877006    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:02:44.877644 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:02:44.877192    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:02:57.882780 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:02:57.882751    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:02:57.883243 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:02:57.882909    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:03:07.932665 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:07.932629    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:03:08.395272 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:08.395219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:10.876761 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:10.876734    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:03:10.877206 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:03:10.876899    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:03:18.394468 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:18.394428    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:22.956031 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:22.955978    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" containerID="cri-o://49a516ee319505e0d48dc7e857ddb517ecc7c6748c2a6c1a2564e1d7a3f7226a" gracePeriod=45
Sep 18 23:03:23.876498 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:23.876463    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:03:23.876716 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:03:23.876643    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:03:28.394638 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:28.394599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:28.395009 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:28.394717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:03:30.575019 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:30.574985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:03:36.877251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:36.877223    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:03:36.877653 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:03:36.877387    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:03:37.622869 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:37.622833    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:38.394371 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:38.394330    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:45.594805 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:45.594766    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main" containerID="cri-o://a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5" gracePeriod=45
Sep 18 23:03:47.622559 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:47.622518    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:47.876938 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:47.876849    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:03:47.877101 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:03:47.877049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:03:48.394891 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:48.394855    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:53.177911 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.177877    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d12a76d-e366-400b-a54d-81bad65b324a" containerID="49a516ee319505e0d48dc7e857ddb517ecc7c6748c2a6c1a2564e1d7a3f7226a" exitCode=0
Sep 18 23:03:53.178366 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.177952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" event={"ID":"9d12a76d-e366-400b-a54d-81bad65b324a","Type":"ContainerDied","Data":"49a516ee319505e0d48dc7e857ddb517ecc7c6748c2a6c1a2564e1d7a3f7226a"}
Sep 18 23:03:53.199225 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.199204    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:03:53.235742 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.235676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs\") pod \"9d12a76d-e366-400b-a54d-81bad65b324a\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") "
Sep 18 23:03:53.235742 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.235728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4clww\" (UniqueName: \"kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww\") pod \"9d12a76d-e366-400b-a54d-81bad65b324a\" (UID: \"9d12a76d-e366-400b-a54d-81bad65b324a\") "
Sep 18 23:03:53.237681 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.237650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww" (OuterVolumeSpecName: "kube-api-access-4clww") pod "9d12a76d-e366-400b-a54d-81bad65b324a" (UID: "9d12a76d-e366-400b-a54d-81bad65b324a"). InnerVolumeSpecName "kube-api-access-4clww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:03:53.237784 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.237729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d12a76d-e366-400b-a54d-81bad65b324a" (UID: "9d12a76d-e366-400b-a54d-81bad65b324a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:03:53.337315 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.337283    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d12a76d-e366-400b-a54d-81bad65b324a-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:03:53.337315 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:53.337312    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4clww\" (UniqueName: \"kubernetes.io/projected/9d12a76d-e366-400b-a54d-81bad65b324a-kube-api-access-4clww\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:03:54.183673 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:54.183598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"
Sep 18 23:03:54.183673 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:54.183601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk" event={"ID":"9d12a76d-e366-400b-a54d-81bad65b324a","Type":"ContainerDied","Data":"99d7c3e3f51ee1bc9f11680c33b766e19ede46f02dfaa1910d177219272a876a"}
Sep 18 23:03:54.183673 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:54.183662    2563 scope.go:117] "RemoveContainer" containerID="49a516ee319505e0d48dc7e857ddb517ecc7c6748c2a6c1a2564e1d7a3f7226a"
Sep 18 23:03:54.204631 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:54.204603    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:03:54.209944 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:54.209922    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-59f747c6b9-hlwjk"]
Sep 18 23:03:55.882204 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:55.882174    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" path="/var/lib/kubelet/pods/9d12a76d-e366-400b-a54d-81bad65b324a/volumes"
Sep 18 23:03:57.622659 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:57.622616    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:03:57.623019 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:03:57.622709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:04:02.876933 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:02.876902    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:04:02.877294 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:02.877079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:04:07.622980 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:07.622947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:04:13.876739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:13.876708    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:04:13.877086 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:13.876878    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:04:15.841855 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:15.841830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:04:15.994143 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:15.994056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs\") pod \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") "
Sep 18 23:04:15.994143 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:15.994131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-425lm\" (UniqueName: \"kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm\") pod \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\" (UID: \"177ecbad-a069-4e9c-a463-ec2a1e10d1db\") "
Sep 18 23:04:15.996327 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:15.996301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "177ecbad-a069-4e9c-a463-ec2a1e10d1db" (UID: "177ecbad-a069-4e9c-a463-ec2a1e10d1db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:04:15.996440 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:15.996300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm" (OuterVolumeSpecName: "kube-api-access-425lm") pod "177ecbad-a069-4e9c-a463-ec2a1e10d1db" (UID: "177ecbad-a069-4e9c-a463-ec2a1e10d1db"). InnerVolumeSpecName "kube-api-access-425lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:04:16.094785 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.094759    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/177ecbad-a069-4e9c-a463-ec2a1e10d1db-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:04:16.094785 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.094784    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425lm\" (UniqueName: \"kubernetes.io/projected/177ecbad-a069-4e9c-a463-ec2a1e10d1db-kube-api-access-425lm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:04:16.281420 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.281310    2563 generic.go:356] "Generic (PLEG): container finished" podID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerID="a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5" exitCode=0
Sep 18 23:04:16.281420 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.281371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"
Sep 18 23:04:16.281611 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.281407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" event={"ID":"177ecbad-a069-4e9c-a463-ec2a1e10d1db","Type":"ContainerDied","Data":"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"}
Sep 18 23:04:16.281611 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.281469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz" event={"ID":"177ecbad-a069-4e9c-a463-ec2a1e10d1db","Type":"ContainerDied","Data":"e47d7614f43a93dbb9a59e89eb42a87075eaf37c973f7bd0c27081bb33b59c71"}
Sep 18 23:04:16.281611 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.281491    2563 scope.go:117] "RemoveContainer" containerID="a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"
Sep 18 23:04:16.290303 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.290282    2563 scope.go:117] "RemoveContainer" containerID="a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"
Sep 18 23:04:16.290583 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:16.290566    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5\": container with ID starting with a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5 not found: ID does not exist" containerID="a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"
Sep 18 23:04:16.290645 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.290591    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5"} err="failed to get container status \"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5\": rpc error: code = NotFound desc = could not find container \"a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5\": container with ID starting with a1e992a818c5a2fff4b191293b11df3175457a730401509f3c6112bc08bde3e5 not found: ID does not exist"
Sep 18 23:04:16.302832 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.302809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:04:16.306270 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:16.306248    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7hbsz"]
Sep 18 23:04:17.882538 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:17.882500    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" path="/var/lib/kubelet/pods/177ecbad-a069-4e9c-a463-ec2a1e10d1db/volumes"
Sep 18 23:04:25.876892 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:25.876863    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:04:25.877275 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:25.877026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:04:36.876755 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:36.876720    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:04:36.877138 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:36.876870    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:04:47.917410 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917363    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:04:47.917791 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917679    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main"
Sep 18 23:04:47.917791 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main"
Sep 18 23:04:47.917791 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main"
Sep 18 23:04:47.917791 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main"
Sep 18 23:04:47.917927 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d12a76d-e366-400b-a54d-81bad65b324a" containerName="main"
Sep 18 23:04:47.917927 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.917808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="177ecbad-a069-4e9c-a463-ec2a1e10d1db" containerName="main"
Sep 18 23:04:47.920413 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.920376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:47.922797 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.922772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 23:04:47.923863 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.923839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 23:04:47.923991 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.923844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 23:04:47.923991 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.923891    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-dc798\""
Sep 18 23:04:47.933229 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:47.933206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:04:48.032007 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.031979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.032159 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.032034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.032159 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.032081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmvq\" (UniqueName: \"kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.032159 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.032102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.032301 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.032184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.032301 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.032217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132561 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmvq\" (UniqueName: \"kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.132712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.132694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.133070 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.133044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.133070 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.133063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.133219 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.133147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.134962 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.134936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.135117 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.135103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.145575 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.145556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmvq\" (UniqueName: \"kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq\") pod \"scheduler-inline-config-test-kserve-8655bc757-7mhqc\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.218490 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.218429    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:04:48.221675 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.221660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.224020 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.224000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-n4m95\""
Sep 18 23:04:48.230588 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.230568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:48.232786 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.232766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:04:48.333539 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.333504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zggq\" (UniqueName: \"kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.333673 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.333560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.434921 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.434891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zggq\" (UniqueName: \"kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.435047 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.434935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.437311 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.437282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.443368 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.443346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zggq\" (UniqueName: \"kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq\") pod \"scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.505520 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.505496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:04:48.507715 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:04:48.507684    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e432a0d_bfc6_4b46_9530_4a50566a5df6.slice/crio-72ca4735d89c6a957e9c7770553ecdcfef474f65bc290f40081f63a7c818fa68 WatchSource:0}: Error finding container 72ca4735d89c6a957e9c7770553ecdcfef474f65bc290f40081f63a7c818fa68: Status 404 returned error can't find the container with id 72ca4735d89c6a957e9c7770553ecdcfef474f65bc290f40081f63a7c818fa68
Sep 18 23:04:48.532923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.532902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:48.821032 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.820952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:04:48.824920 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:04:48.824894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e0b446_29c4_422e_b155_95de617ba04b.slice/crio-7ab5193f7ecf829064ecf80999a56b5059000c79088f01c2b87a67952329d11c WatchSource:0}: Error finding container 7ab5193f7ecf829064ecf80999a56b5059000c79088f01c2b87a67952329d11c: Status 404 returned error can't find the container with id 7ab5193f7ecf829064ecf80999a56b5059000c79088f01c2b87a67952329d11c
Sep 18 23:04:48.876797 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:48.876778    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:04:48.876953 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:04:48.876935    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:04:49.429470 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.429432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" event={"ID":"8e432a0d-bfc6-4b46-9530-4a50566a5df6","Type":"ContainerStarted","Data":"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"}
Sep 18 23:04:49.429879 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.429536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" event={"ID":"8e432a0d-bfc6-4b46-9530-4a50566a5df6","Type":"ContainerStarted","Data":"72ca4735d89c6a957e9c7770553ecdcfef474f65bc290f40081f63a7c818fa68"}
Sep 18 23:04:49.430941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.430918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" event={"ID":"22e0b446-29c4-422e-b155-95de617ba04b","Type":"ContainerStarted","Data":"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"}
Sep 18 23:04:49.431020 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.430966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:04:49.431020 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.430983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" event={"ID":"22e0b446-29c4-422e-b155-95de617ba04b","Type":"ContainerStarted","Data":"7ab5193f7ecf829064ecf80999a56b5059000c79088f01c2b87a67952329d11c"}
Sep 18 23:04:49.448459 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.448416    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" podStartSLOduration=2.448385752 podStartE2EDuration="2.448385752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:04:49.447202575 +0000 UTC m=+4429.645910888" watchObservedRunningTime="2026-09-18 23:04:49.448385752 +0000 UTC m=+4429.647094051"
Sep 18 23:04:49.465495 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:49.465446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podStartSLOduration=1.465428998 podStartE2EDuration="1.465428998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:04:49.463370912 +0000 UTC m=+4429.662079213" watchObservedRunningTime="2026-09-18 23:04:49.465428998 +0000 UTC m=+4429.664137298"
Sep 18 23:04:50.231298 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:50.231266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:58.230954 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:58.230918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:58.243520 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:58.243495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:04:58.483305 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:04:58.483229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:05:02.876702 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:02.876671    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:05:02.877106 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:02.876852    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:14.876474 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:14.876438    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:05:14.876839 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:14.876688    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:20.436789 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:20.436756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:05:21.915539 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:21.915506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:05:21.918522 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:21.918494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:05:21.927842 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:21.927810    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 23:05:21.927842 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:05:21.927842 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:05:21.927842 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f" cmd=["/bin/sleep","15"]
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:21.927857    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" message="" err=<
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:21.927869    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerName="main" containerID="cri-o://7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"
Sep 18 23:05:21.928144 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:21.927898    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerName="main" containerID="cri-o://7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f" gracePeriod=60
Sep 18 23:05:22.274334 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.274312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:05:22.389636 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzmvq\" (UniqueName: \"kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389792 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389792 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389792 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389792 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache\") pod \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\" (UID: \"8e432a0d-bfc6-4b46-9530-4a50566a5df6\") "
Sep 18 23:05:22.389997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.389952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:05:22.390115 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.390054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home" (OuterVolumeSpecName: "home") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:05:22.390115 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.390096    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.390223 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.390114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:05:22.391912 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.391888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq" (OuterVolumeSpecName: "kube-api-access-xzmvq") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "kube-api-access-xzmvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:05:22.391912 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.391896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm" (OuterVolumeSpecName: "dshm") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:05:22.392046 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.391952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e432a0d-bfc6-4b46-9530-4a50566a5df6" (UID: "8e432a0d-bfc6-4b46-9530-4a50566a5df6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:05:22.490923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.490862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzmvq\" (UniqueName: \"kubernetes.io/projected/8e432a0d-bfc6-4b46-9530-4a50566a5df6-kube-api-access-xzmvq\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.490923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.490886    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.490923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.490895    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e432a0d-bfc6-4b46-9530-4a50566a5df6-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.490923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.490904    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.490923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.490911    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e432a0d-bfc6-4b46-9530-4a50566a5df6-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:05:22.586807 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.586775    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerID="7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f" exitCode=0
Sep 18 23:05:22.586941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.586815    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"
Sep 18 23:05:22.586941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.586859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" event={"ID":"8e432a0d-bfc6-4b46-9530-4a50566a5df6","Type":"ContainerDied","Data":"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"}
Sep 18 23:05:22.586941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.586911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc" event={"ID":"8e432a0d-bfc6-4b46-9530-4a50566a5df6","Type":"ContainerDied","Data":"72ca4735d89c6a957e9c7770553ecdcfef474f65bc290f40081f63a7c818fa68"}
Sep 18 23:05:22.586941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.586938    2563 scope.go:117] "RemoveContainer" containerID="7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"
Sep 18 23:05:22.597186 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.597166    2563 scope.go:117] "RemoveContainer" containerID="7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"
Sep 18 23:05:22.597495 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:22.597476    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f\": container with ID starting with 7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f not found: ID does not exist" containerID="7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"
Sep 18 23:05:22.597626 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.597504    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f"} err="failed to get container status \"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f\": rpc error: code = NotFound desc = could not find container \"7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f\": container with ID starting with 7dcf0a4a9f568a8ad9b09982be83f5b57a684ca99eb7e250bb5069b4e1340d6f not found: ID does not exist"
Sep 18 23:05:22.610258 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.610229    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:05:22.615221 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:22.615198    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-7mhqc"]
Sep 18 23:05:23.881853 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:23.881824    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" path="/var/lib/kubelet/pods/8e432a0d-bfc6-4b46-9530-4a50566a5df6/volumes"
Sep 18 23:05:26.876681 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:26.876655    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:05:27.612628 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.612601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:05:27.613046 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.613030    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/15.log"
Sep 18 23:05:27.613109 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.613064    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69" exitCode=1
Sep 18 23:05:27.613109 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.613099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"}
Sep 18 23:05:27.613179 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.613145    2563 scope.go:117] "RemoveContainer" containerID="f79bcdf748ad8a1dc9f6487e9cc57c73d150ee731bd5ca0aff834d801f6ccab6"
Sep 18 23:05:27.613457 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:27.613432    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:05:27.613615 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:27.613595    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:28.618857 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:28.618830    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:05:30.437667 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:30.437630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:05:34.301953 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:34.301918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:05:34.302536 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:34.302299    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:05:34.321025 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:34.302538    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:34.964624 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:34.964590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:05:34.964908 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:34.964895    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:05:34.965051 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:34.965034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:36.937510 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:36.937460    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main" containerID="cri-o://f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089" gracePeriod=45
Sep 18 23:05:40.437027 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:40.436987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:05:47.877310 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:47.877220    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:05:47.877783 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:47.877509    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:05:50.438025 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:50.437974    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:05:50.438585 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:50.438143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:05:59.877122 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:05:59.877091    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:05:59.877617 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:05:59.877246    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:06:00.437489 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:00.437446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:06:02.197987 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:02.197959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:06:02.210238 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:02.210218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:06:02.210578 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:02.210555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:06:02.221244 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:02.221219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:06:07.180051 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.180027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:06:07.323644 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.323566    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zggq\" (UniqueName: \"kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq\") pod \"22e0b446-29c4-422e-b155-95de617ba04b\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") "
Sep 18 23:06:07.323644 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.323629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs\") pod \"22e0b446-29c4-422e-b155-95de617ba04b\" (UID: \"22e0b446-29c4-422e-b155-95de617ba04b\") "
Sep 18 23:06:07.325712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.325684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq" (OuterVolumeSpecName: "kube-api-access-7zggq") pod "22e0b446-29c4-422e-b155-95de617ba04b" (UID: "22e0b446-29c4-422e-b155-95de617ba04b"). InnerVolumeSpecName "kube-api-access-7zggq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:06:07.325812 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.325681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22e0b446-29c4-422e-b155-95de617ba04b" (UID: "22e0b446-29c4-422e-b155-95de617ba04b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:06:07.424519 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.424490    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zggq\" (UniqueName: \"kubernetes.io/projected/22e0b446-29c4-422e-b155-95de617ba04b-kube-api-access-7zggq\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:07.424519 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.424520    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e0b446-29c4-422e-b155-95de617ba04b-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:07.794263 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.794230    2563 generic.go:356] "Generic (PLEG): container finished" podID="22e0b446-29c4-422e-b155-95de617ba04b" containerID="f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089" exitCode=0
Sep 18 23:06:07.794444 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.794271    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"
Sep 18 23:06:07.794444 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.794304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" event={"ID":"22e0b446-29c4-422e-b155-95de617ba04b","Type":"ContainerDied","Data":"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"}
Sep 18 23:06:07.794444 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.794344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl" event={"ID":"22e0b446-29c4-422e-b155-95de617ba04b","Type":"ContainerDied","Data":"7ab5193f7ecf829064ecf80999a56b5059000c79088f01c2b87a67952329d11c"}
Sep 18 23:06:07.794444 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.794367    2563 scope.go:117] "RemoveContainer" containerID="f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"
Sep 18 23:06:07.802730 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.802710    2563 scope.go:117] "RemoveContainer" containerID="f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"
Sep 18 23:06:07.802970 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:07.802952    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089\": container with ID starting with f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089 not found: ID does not exist" containerID="f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"
Sep 18 23:06:07.803029 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.802977    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089"} err="failed to get container status \"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089\": rpc error: code = NotFound desc = could not find container \"f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089\": container with ID starting with f417544e11cd4d470674ed4a3135bdba3298ca8ac84a5440887329d07cc81089 not found: ID does not exist"
Sep 18 23:06:07.820810 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.820177    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:06:07.820810 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.820225    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-859b5fffrl"]
Sep 18 23:06:07.883121 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:07.883097    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e0b446-29c4-422e-b155-95de617ba04b" path="/var/lib/kubelet/pods/22e0b446-29c4-422e-b155-95de617ba04b/volumes"
Sep 18 23:06:13.876867 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:13.876835    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:06:13.877329 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:13.877049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:06:16.496267 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerName="main"
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerName="main"
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496602    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main"
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496609    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main"
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496654    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e432a0d-bfc6-4b46-9530-4a50566a5df6" containerName="main"
Sep 18 23:06:16.496658 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.496662    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e0b446-29c4-422e-b155-95de617ba04b" containerName="main"
Sep 18 23:06:16.499527 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.499506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.503345 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.503324    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 23:06:16.503484 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.503364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 23:06:16.503484 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.503382    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 23:06:16.503739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.503722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-vp5jm\""
Sep 18 23:06:16.511008 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.510986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:16.591788 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.591903 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.591903 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.591995 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.591995 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.591995 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.591950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpvk\" (UniqueName: \"kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692554 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692664 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692664 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692740 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692782 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.692904 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpvk\" (UniqueName: \"kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.693030 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.692881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.693095 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.693052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.693148 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.693090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.694895 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.694878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.695026 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.695009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.702083 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.702063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpvk\" (UniqueName: \"kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.793740 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.793672    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:06:16.797021 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.797003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:16.809370 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.809346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:16.827976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.827955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-89sn7\""
Sep 18 23:06:16.896754 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.896671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v9d\" (UniqueName: \"kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:16.897537 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.896814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:16.899107 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.899081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:06:16.997368 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.997342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:16.997521 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.997412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v9d\" (UniqueName: \"kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:16.999978 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:16.999956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:17.019898 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.019875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v9d\" (UniqueName: \"kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:17.107353 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.107327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:17.114219 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.114123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:17.117239 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:06:17.117174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f635e0d_dbd9_4fe8_875f_bc45a6a3a0a5.slice/crio-a9d16e1c3795646f73708050a5e78fe0496566b99345380e2734a33fa7e3a2d8 WatchSource:0}: Error finding container a9d16e1c3795646f73708050a5e78fe0496566b99345380e2734a33fa7e3a2d8: Status 404 returned error can't find the container with id a9d16e1c3795646f73708050a5e78fe0496566b99345380e2734a33fa7e3a2d8
Sep 18 23:06:17.404815 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:06:17.404786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc482fe5_6252_4201_9350_6846cfc2d82f.slice/crio-e07e3d8dad47ce841479e75e839a60db86047284190d450649f211f33d2ec4ef WatchSource:0}: Error finding container e07e3d8dad47ce841479e75e839a60db86047284190d450649f211f33d2ec4ef: Status 404 returned error can't find the container with id e07e3d8dad47ce841479e75e839a60db86047284190d450649f211f33d2ec4ef
Sep 18 23:06:17.406367 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.406337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:06:17.851257 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.851219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" event={"ID":"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5","Type":"ContainerStarted","Data":"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"}
Sep 18 23:06:17.851725 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.851286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" event={"ID":"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5","Type":"ContainerStarted","Data":"a9d16e1c3795646f73708050a5e78fe0496566b99345380e2734a33fa7e3a2d8"}
Sep 18 23:06:17.853075 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.853050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" event={"ID":"fc482fe5-6252-4201-9350-6846cfc2d82f","Type":"ContainerStarted","Data":"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"}
Sep 18 23:06:17.853230 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.853091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" event={"ID":"fc482fe5-6252-4201-9350-6846cfc2d82f","Type":"ContainerStarted","Data":"e07e3d8dad47ce841479e75e839a60db86047284190d450649f211f33d2ec4ef"}
Sep 18 23:06:17.853230 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.853171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:17.894827 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.894783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podStartSLOduration=1.894771414 podStartE2EDuration="1.894771414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:06:17.894118083 +0000 UTC m=+4518.092826444" watchObservedRunningTime="2026-09-18 23:06:17.894771414 +0000 UTC m=+4518.093479712"
Sep 18 23:06:17.895172 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:17.895148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" podStartSLOduration=1.8951397509999999 podStartE2EDuration="1.895139751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:06:17.877383736 +0000 UTC m=+4518.076092036" watchObservedRunningTime="2026-09-18 23:06:17.895139751 +0000 UTC m=+4518.093848050"
Sep 18 23:06:18.809447 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:18.809414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:26.809499 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:26.809463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:26.821870 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:26.821838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:26.876928 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:26.876909    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:06:26.877088 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:26.877069    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:06:26.903776 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:26.903752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:39.877489 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:39.877453    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:06:39.877937 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:39.877703    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:06:48.859545 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:48.859515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:06:51.878751 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:51.878720    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:06:51.879249 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:51.878917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:06:57.438261 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.438231    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:57.450753 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.450724    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:06:57.451598 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:57.451539    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 23:06:57.451598 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:06:57.451598 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:06:57.451598 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493" cmd=["/bin/sleep","15"]
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.451616    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" message="" err=<
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:57.451633    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerName="main" containerID="cri-o://bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"
Sep 18 23:06:57.451934 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.451671    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerName="main" containerID="cri-o://bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493" gracePeriod=60
Sep 18 23:06:57.809472 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.809447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:57.908980 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.908954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.908996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzpvk\" (UniqueName: \"kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir\") pod \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\" (UID: \"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5\") "
Sep 18 23:06:57.909456 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home" (OuterVolumeSpecName: "home") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:57.909456 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:57.909456 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909438    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:57.909582 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:57.909582 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.909459    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:57.911307 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.911277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk" (OuterVolumeSpecName: "kube-api-access-bzpvk") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "kube-api-access-bzpvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:06:57.911575 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.911541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm" (OuterVolumeSpecName: "dshm") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:57.911663 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:57.911575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" (UID: "5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:06:58.010997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.010838    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:58.010997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.010873    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:58.010997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.010888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpvk\" (UniqueName: \"kubernetes.io/projected/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-kube-api-access-bzpvk\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:58.010997 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.010901    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:06:58.037241 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.037206    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerID="bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493" exitCode=0
Sep 18 23:06:58.037374 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.037250    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"
Sep 18 23:06:58.037374 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.037282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" event={"ID":"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5","Type":"ContainerDied","Data":"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"}
Sep 18 23:06:58.037374 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.037341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89" event={"ID":"5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5","Type":"ContainerDied","Data":"a9d16e1c3795646f73708050a5e78fe0496566b99345380e2734a33fa7e3a2d8"}
Sep 18 23:06:58.037374 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.037362    2563 scope.go:117] "RemoveContainer" containerID="bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"
Sep 18 23:06:58.047726 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.047681    2563 scope.go:117] "RemoveContainer" containerID="bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"
Sep 18 23:06:58.048061 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:06:58.048040    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493\": container with ID starting with bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493 not found: ID does not exist" containerID="bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"
Sep 18 23:06:58.048149 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.048071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493"} err="failed to get container status \"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493\": rpc error: code = NotFound desc = could not find container \"bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493\": container with ID starting with bfcfe03dd226a09d12c1729321e08cadf3eccc2d6f689c9e8618f3bd612ac493 not found: ID does not exist"
Sep 18 23:06:58.060812 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.060783    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:58.071686 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.068982    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f5pw89"]
Sep 18 23:06:58.858921 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:58.858886    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:06:59.882673 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:06:59.882640    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" path="/var/lib/kubelet/pods/5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5/volumes"
Sep 18 23:07:02.876591 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:02.876565    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:07:02.876956 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:02.876723    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:07:08.859309 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:08.859273    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:07:12.473956 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:12.473910    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" containerID="cri-o://a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964" gracePeriod=45
Sep 18 23:07:15.876875 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:15.876798    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:07:15.877311 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:15.877008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:07:18.859571 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:18.859524    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:07:18.860039 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:18.859640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:07:28.859680 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:28.859638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:07:29.882662 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:29.882632    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:07:29.883036 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:29.882796    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:07:38.859693 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:38.859649    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:07:42.719814 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.719787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:07:42.828654 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.828595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs\") pod \"fc482fe5-6252-4201-9350-6846cfc2d82f\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") "
Sep 18 23:07:42.828764 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.828705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8v9d\" (UniqueName: \"kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d\") pod \"fc482fe5-6252-4201-9350-6846cfc2d82f\" (UID: \"fc482fe5-6252-4201-9350-6846cfc2d82f\") "
Sep 18 23:07:42.830617 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.830587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc482fe5-6252-4201-9350-6846cfc2d82f" (UID: "fc482fe5-6252-4201-9350-6846cfc2d82f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:07:42.830719 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.830672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d" (OuterVolumeSpecName: "kube-api-access-c8v9d") pod "fc482fe5-6252-4201-9350-6846cfc2d82f" (UID: "fc482fe5-6252-4201-9350-6846cfc2d82f"). InnerVolumeSpecName "kube-api-access-c8v9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:07:42.929566 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.929540    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc482fe5-6252-4201-9350-6846cfc2d82f-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:07:42.929679 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:42.929579    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8v9d\" (UniqueName: \"kubernetes.io/projected/fc482fe5-6252-4201-9350-6846cfc2d82f-kube-api-access-c8v9d\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:07:43.239958 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.239922    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerID="a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964" exitCode=0
Sep 18 23:07:43.240124 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.239977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"
Sep 18 23:07:43.240124 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.240006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" event={"ID":"fc482fe5-6252-4201-9350-6846cfc2d82f","Type":"ContainerDied","Data":"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"}
Sep 18 23:07:43.240124 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.240066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp" event={"ID":"fc482fe5-6252-4201-9350-6846cfc2d82f","Type":"ContainerDied","Data":"e07e3d8dad47ce841479e75e839a60db86047284190d450649f211f33d2ec4ef"}
Sep 18 23:07:43.240124 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.240095    2563 scope.go:117] "RemoveContainer" containerID="a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"
Sep 18 23:07:43.251193 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.251177    2563 scope.go:117] "RemoveContainer" containerID="a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"
Sep 18 23:07:43.251558 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:43.251535    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964\": container with ID starting with a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964 not found: ID does not exist" containerID="a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"
Sep 18 23:07:43.251639 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.251569    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964"} err="failed to get container status \"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964\": rpc error: code = NotFound desc = could not find container \"a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964\": container with ID starting with a29bb4a76ca1c7474b761912c7b27cc6cd4363c2710462344ed8367a541c9964 not found: ID does not exist"
Sep 18 23:07:43.267565 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.267541    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:07:43.272439 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.272416    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheskkgp"]
Sep 18 23:07:43.882696 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:43.882665    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" path="/var/lib/kubelet/pods/fc482fe5-6252-4201-9350-6846cfc2d82f/volumes"
Sep 18 23:07:44.877002 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:44.876975    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:07:44.877170 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:44.877125    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:07:54.714781 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.714747    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715056    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerName="main"
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715068    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerName="main"
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715094    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main"
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715103    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main"
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715166    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc482fe5-6252-4201-9350-6846cfc2d82f" containerName="main"
Sep 18 23:07:54.715502 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.715181    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f635e0d-dbd9-4fe8-875f-bc45a6a3a0a5" containerName="main"
Sep 18 23:07:54.718098 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.718080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.720772 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.720754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 23:07:54.720772 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.720763    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 23:07:54.721768 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.721746    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-xr96v\""
Sep 18 23:07:54.721827 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.721811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 23:07:54.728694 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.728673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:07:54.811863 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.811838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.811976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.811881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.811976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.811916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.811976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.811951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.811976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.811968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.812117 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.812036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzdx\" (UniqueName: \"kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912350 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912493 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912493 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzdx\" (UniqueName: \"kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912599 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912599 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912688 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912870 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912957 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.912957 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.912938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.914568 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.914546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.914942 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.914922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:54.921155 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:54.921128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzdx\" (UniqueName: \"kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:55.028602 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.028525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:07:55.029366 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.029333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:07:55.032815 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.032799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.035094 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.035071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-nz5qs\""
Sep 18 23:07:55.041923 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.041902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:07:55.215026 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.214994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.215194 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.215058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrfw\" (UniqueName: \"kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.315553 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.315482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrfw\" (UniqueName: \"kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.315716 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.315612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.318283 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.318263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.319037 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:07:55.319012    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb906ba62_3c6a_4be5_95ba_dcf72c7b0b3c.slice/crio-6333466a54edd3287d097a584cd301090d3931edb72bd1061202ec34c9424ea6 WatchSource:0}: Error finding container 6333466a54edd3287d097a584cd301090d3931edb72bd1061202ec34c9424ea6: Status 404 returned error can't find the container with id 6333466a54edd3287d097a584cd301090d3931edb72bd1061202ec34c9424ea6
Sep 18 23:07:55.319143 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.319080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:07:55.328857 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.328839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrfw\" (UniqueName: \"kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.359348 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.359327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:55.634586 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.634550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:07:55.637053 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:07:55.637027    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da8bc89_3fd5_4724_a75b_25aafeba3d8f.slice/crio-ffe3e13ccc0c80fd7fa9a56144e787886268c1c8a364bc660a6bdd30b288ec84 WatchSource:0}: Error finding container ffe3e13ccc0c80fd7fa9a56144e787886268c1c8a364bc660a6bdd30b288ec84: Status 404 returned error can't find the container with id ffe3e13ccc0c80fd7fa9a56144e787886268c1c8a364bc660a6bdd30b288ec84
Sep 18 23:07:55.877182 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:55.877148    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:07:55.877592 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:07:55.877379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:07:56.302427 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.302378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" event={"ID":"4da8bc89-3fd5-4724-a75b-25aafeba3d8f","Type":"ContainerStarted","Data":"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"}
Sep 18 23:07:56.302643 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.302441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" event={"ID":"4da8bc89-3fd5-4724-a75b-25aafeba3d8f","Type":"ContainerStarted","Data":"ffe3e13ccc0c80fd7fa9a56144e787886268c1c8a364bc660a6bdd30b288ec84"}
Sep 18 23:07:56.302643 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.302462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:07:56.304020 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.303995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" event={"ID":"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c","Type":"ContainerStarted","Data":"449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a"}
Sep 18 23:07:56.304127 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.304035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" event={"ID":"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c","Type":"ContainerStarted","Data":"6333466a54edd3287d097a584cd301090d3931edb72bd1061202ec34c9424ea6"}
Sep 18 23:07:56.319783 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.319741    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podStartSLOduration=1.319727514 podStartE2EDuration="1.319727514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:07:56.31892241 +0000 UTC m=+4616.517630710" watchObservedRunningTime="2026-09-18 23:07:56.319727514 +0000 UTC m=+4616.518435804"
Sep 18 23:07:56.338199 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:56.338161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" podStartSLOduration=2.3381490129999998 podStartE2EDuration="2.338149013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:07:56.336701651 +0000 UTC m=+4616.535409963" watchObservedRunningTime="2026-09-18 23:07:56.338149013 +0000 UTC m=+4616.536857317"
Sep 18 23:07:57.029571 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:07:57.029535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:05.029001 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:05.028967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:05.041574 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:05.041546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:05.353180 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:05.353151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:07.876560 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:07.876531    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:08:07.877022 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:07.876762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:08:21.878982 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:21.878942    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:08:21.879430 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:21.879161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:08:27.309773 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:27.309742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:08:28.937562 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:28.937525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:08:28.946163 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:28.946050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:28.948606    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a" cmd=["/bin/sleep","15"]
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:28.948661    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" message="" err=<
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:28.948679    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main" containerID="cri-o://449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a"
Sep 18 23:08:28.949195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:28.948713    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main" containerID="cri-o://449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a" gracePeriod=60
Sep 18 23:08:29.342966 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.342931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 18 23:08:29.450634 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.450608    2563 generic.go:356] "Generic (PLEG): container finished" podID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerID="449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a" exitCode=0
Sep 18 23:08:29.450790 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.450660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" event={"ID":"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c","Type":"ContainerDied","Data":"449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a"}
Sep 18 23:08:29.712373 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.710023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:29.773800 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.773769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.773800 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.773803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.773996 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.773855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.773996 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.773893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.774106 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.774106 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hzdx\" (UniqueName: \"kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx\") pod \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\" (UID: \"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c\") "
Sep 18 23:08:29.774106 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:29.774254 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:29.774862 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home" (OuterVolumeSpecName: "home") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:29.775004 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774944    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:29.775004 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.774967    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:29.776782 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.776752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm" (OuterVolumeSpecName: "dshm") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:29.783141 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.783108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx" (OuterVolumeSpecName: "kube-api-access-4hzdx") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "kube-api-access-4hzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:08:29.784341 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.784178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" (UID: "b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:08:29.875600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.875532    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:29.875600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.875559    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:29.875600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.875569    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:29.875600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:29.875599    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzdx\" (UniqueName: \"kubernetes.io/projected/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c-kube-api-access-4hzdx\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:08:30.456968 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:30.456926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r" event={"ID":"b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c","Type":"ContainerDied","Data":"6333466a54edd3287d097a584cd301090d3931edb72bd1061202ec34c9424ea6"}
Sep 18 23:08:30.457383 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:30.457004    2563 scope.go:117] "RemoveContainer" containerID="449ff3c6b6d605d9511bfc57fc7548f89bd9c9647fb6c060c305c9a5ec1e2f3a"
Sep 18 23:08:30.457383 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:30.456948    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"
Sep 18 23:08:30.475653 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:30.475620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:08:30.478854 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:30.478831    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-gdx4r"]
Sep 18 23:08:31.883053 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:31.883022    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" path="/var/lib/kubelet/pods/b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c/volumes"
Sep 18 23:08:33.876666 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:33.876635    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:08:33.877046 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:33.876790    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:08:37.309932 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:37.309893    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:08:43.967456 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:43.967421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main" containerID="cri-o://cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c" gracePeriod=45
Sep 18 23:08:44.876826 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:44.876747    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:08:44.876981 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:44.876934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:08:47.309776 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:47.309736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:08:57.309758 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:57.309719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:08:57.310137 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:57.309816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:08:58.877070 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:08:58.877041    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:08:58.877538 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:08:58.877202    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:09:07.309749 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:07.309707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:09:10.877152 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:10.877122    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:09:10.877618 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:09:10.877302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:09:14.219707 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.219677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:09:14.397638 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.397598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzrfw\" (UniqueName: \"kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw\") pod \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") "
Sep 18 23:09:14.397825 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.397692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs\") pod \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\" (UID: \"4da8bc89-3fd5-4724-a75b-25aafeba3d8f\") "
Sep 18 23:09:14.399794 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.399759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4da8bc89-3fd5-4724-a75b-25aafeba3d8f" (UID: "4da8bc89-3fd5-4724-a75b-25aafeba3d8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:09:14.399913 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.399822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw" (OuterVolumeSpecName: "kube-api-access-tzrfw") pod "4da8bc89-3fd5-4724-a75b-25aafeba3d8f" (UID: "4da8bc89-3fd5-4724-a75b-25aafeba3d8f"). InnerVolumeSpecName "kube-api-access-tzrfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:09:14.498471 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.498438    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:09:14.498471 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.498466    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrfw\" (UniqueName: \"kubernetes.io/projected/4da8bc89-3fd5-4724-a75b-25aafeba3d8f-kube-api-access-tzrfw\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:09:14.661338 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.661248    2563 generic.go:356] "Generic (PLEG): container finished" podID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerID="cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c" exitCode=0
Sep 18 23:09:14.661338 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.661297    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"
Sep 18 23:09:14.661548 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.661298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" event={"ID":"4da8bc89-3fd5-4724-a75b-25aafeba3d8f","Type":"ContainerDied","Data":"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"}
Sep 18 23:09:14.661548 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.661429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l" event={"ID":"4da8bc89-3fd5-4724-a75b-25aafeba3d8f","Type":"ContainerDied","Data":"ffe3e13ccc0c80fd7fa9a56144e787886268c1c8a364bc660a6bdd30b288ec84"}
Sep 18 23:09:14.661548 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.661461    2563 scope.go:117] "RemoveContainer" containerID="cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"
Sep 18 23:09:14.671903 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.671534    2563 scope.go:117] "RemoveContainer" containerID="cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"
Sep 18 23:09:14.672224 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:09:14.672203    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c\": container with ID starting with cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c not found: ID does not exist" containerID="cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"
Sep 18 23:09:14.672320 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.672231    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c"} err="failed to get container status \"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c\": rpc error: code = NotFound desc = could not find container \"cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c\": container with ID starting with cebb1fd081d2337d4bc7ffe299feeb68b7d5056e120f210c320b31552505b48c not found: ID does not exist"
Sep 18 23:09:14.684667 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.684642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:09:14.690434 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:14.690413    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-67c6cfq25l"]
Sep 18 23:09:15.881550 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:15.881509    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" path="/var/lib/kubelet/pods/4da8bc89-3fd5-4724-a75b-25aafeba3d8f/volumes"
Sep 18 23:09:21.881488 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:21.881456    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:09:21.881929 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:09:21.881635    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:09:35.876671 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:35.876637    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:09:35.877130 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:09:35.876848    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:09:45.995968 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.995932    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main"
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996410    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main"
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main"
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996449    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main"
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996550    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4da8bc89-3fd5-4724-a75b-25aafeba3d8f" containerName="main"
Sep 18 23:09:45.996573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.996565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b906ba62-3c6a-4be5-95ba-dcf72c7b0b3c" containerName="main"
Sep 18 23:09:45.999479 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:45.999457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.002674 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.002651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-epp-sa-dockercfg-ng865\""
Sep 18 23:09:46.002783 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.002714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 18 23:09:46.002783 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.002745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 23:09:46.002906 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.002829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-pdcfl\""
Sep 18 23:09:46.002906 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.002880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 18 23:09:46.007549 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.007528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:09:46.018038 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.018010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.018150 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.018083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlht4\" (UniqueName: \"kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.119320 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.119295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlht4\" (UniqueName: \"kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.119476 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.119354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.121767 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.121741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.128613 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.128589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlht4\" (UniqueName: \"kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4\") pod \"tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.310193 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.310118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.584099 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.584071    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:09:46.586328 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:09:46.586296    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5794c3e9_efb3_451c_9257_94778463c017.slice/crio-2266b8954a50f91de3ca31d06250d9fbaa73b8c4c4ae7ba15a47eb49c9d2e6f7 WatchSource:0}: Error finding container 2266b8954a50f91de3ca31d06250d9fbaa73b8c4c4ae7ba15a47eb49c9d2e6f7: Status 404 returned error can't find the container with id 2266b8954a50f91de3ca31d06250d9fbaa73b8c4c4ae7ba15a47eb49c9d2e6f7
Sep 18 23:09:46.803857 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.803809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" event={"ID":"5794c3e9-efb3-451c-9257-94778463c017","Type":"ContainerStarted","Data":"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"}
Sep 18 23:09:46.804042 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.803874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" event={"ID":"5794c3e9-efb3-451c-9257-94778463c017","Type":"ContainerStarted","Data":"2266b8954a50f91de3ca31d06250d9fbaa73b8c4c4ae7ba15a47eb49c9d2e6f7"}
Sep 18 23:09:46.804042 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.803952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:09:46.822520 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:46.822458    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podStartSLOduration=1.822443316 podStartE2EDuration="1.822443316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:09:46.821709065 +0000 UTC m=+4727.020417365" watchObservedRunningTime="2026-09-18 23:09:46.822443316 +0000 UTC m=+4727.021151615"
Sep 18 23:09:50.876779 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:09:50.876749    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:09:50.877148 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:09:50.876913    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:03.876751 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:03.876722    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:10:03.877130 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:03.876898    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:15.876460 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:15.876374    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:10:15.876921 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:15.876597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:17.810258 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:17.810229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:10:27.876639 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:27.876604    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:10:27.982498 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:27.982477    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:10:27.982608 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:27.982543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerStarted","Data":"4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"}
Sep 18 23:10:27.982755 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:27.982735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:10:28.988694 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.988667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/17.log"
Sep 18 23:10:28.989179 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.989162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/16.log"
Sep 18 23:10:28.989224 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.989198    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee" exitCode=1
Sep 18 23:10:28.989260 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.989230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"}
Sep 18 23:10:28.989296 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.989289    2563 scope.go:117] "RemoveContainer" containerID="c51062ffb0d4171928f9e8dc6371ab07fb8f19df3e696af77e502805e2efbb69"
Sep 18 23:10:28.989515 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:28.989491    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:10:28.989709 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:28.989685    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:29.994765 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:29.994736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/17.log"
Sep 18 23:10:29.995157 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:29.995020    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:10:29.995203 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:29.995170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:34.301651 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:34.301617    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:10:34.302036 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:34.301923    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:10:34.302102 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:34.302083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:48.876356 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:48.876318    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:10:48.879087 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:48.876513    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:10:59.877313 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:10:59.877279    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:10:59.877748 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:10:59.877483    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:11:02.223779 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:02.223751    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/17.log"
Sep 18 23:11:02.235854 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:02.235832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:11:02.236121 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:02.236101    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/17.log"
Sep 18 23:11:02.246912 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:02.246896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:11:13.876608 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:13.876575    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:11:13.876994 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:11:13.876732    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:11:28.877004 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:28.876970    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:11:28.877486 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:11:28.877126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:11:39.876577 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:39.876546    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:11:39.876945 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:11:39.876702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:11:50.876894 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:11:50.876803    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:11:50.877312 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:11:50.877023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:12:04.876322 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:12:04.876290    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:12:04.876762 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:12:04.876491    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:12:15.877112 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:12:15.877076    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:12:15.877558 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:12:15.877256    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:12:30.876715 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:12:30.876687    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:12:30.877183 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:12:30.876843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:12:42.877273 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:12:42.877242    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:12:42.877765 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:12:42.877439    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:12:56.877075 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:12:56.877035    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:12:56.877551 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:12:56.877262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:13:00.615262 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.615230    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:13:00.618421 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.618372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.620753 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.620732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 23:13:00.620898 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.620872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 23:13:00.621296 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.621272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-9nwvf\""
Sep 18 23:13:00.621591 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.621575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 18 23:13:00.629688 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.629665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:13:00.656745 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.656723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.656848 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.656788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.656848 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.656813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.656848 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.656840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.657026 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.656950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhrrf\" (UniqueName: \"kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.657026 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.657008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.757911 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.757885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.758039 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.757933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhrrf\" (UniqueName: \"kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.758039 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.757960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.758039 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.757983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.758039 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.758023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.758272 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.758048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.759141 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.758539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.759141 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.758811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.759141 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.759013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.760986 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.760965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.761284 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.761250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:00.766290 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:00.766266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhrrf\" (UniqueName: \"kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.532590 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.532558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.537712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.537689    2563 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.633270 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.633240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.633412 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.633372    2563 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.636361 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.636343    2563 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") DevicePath \"csi-6fc887ba541289b651f973da1d0d25c46b17eaa86304eabb6cbb830ba08f7559\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.734445 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.734413    2563 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9f88c7dcb205a4ab1e755ef598d6ac0d3322c3e055fcdfe662aa08fc65c6ffbd/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.753264 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.753242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"lora-pvc-adapters-test-kserve-66f88c7f48-jch2d\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.832600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.832546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-9nwvf\""
Sep 18 23:13:10.840468 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.840449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:10.876525 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:10.876492    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:13:10.876732 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:13:10.876711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:13:11.121505 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:11.121475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:13:11.124261 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:13:11.124233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0889743_6d13_4c7f_be6c_e9976a928fe1.slice/crio-77d9cfac9cc13495e95a73827eafca06d82d36c3bf18fa11ddb7faf8cfa2e9bc WatchSource:0}: Error finding container 77d9cfac9cc13495e95a73827eafca06d82d36c3bf18fa11ddb7faf8cfa2e9bc: Status 404 returned error can't find the container with id 77d9cfac9cc13495e95a73827eafca06d82d36c3bf18fa11ddb7faf8cfa2e9bc
Sep 18 23:13:11.713938 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:11.713901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" event={"ID":"f0889743-6d13-4c7f-be6c-e9976a928fe1","Type":"ContainerStarted","Data":"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"}
Sep 18 23:13:11.714389 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:11.713954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" event={"ID":"f0889743-6d13-4c7f-be6c-e9976a928fe1","Type":"ContainerStarted","Data":"77d9cfac9cc13495e95a73827eafca06d82d36c3bf18fa11ddb7faf8cfa2e9bc"}
Sep 18 23:13:11.736619 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:11.736569    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podStartSLOduration=11.736553803 podStartE2EDuration="11.736553803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:13:11.734133612 +0000 UTC m=+4931.932841911" watchObservedRunningTime="2026-09-18 23:13:11.736553803 +0000 UTC m=+4931.935262102"
Sep 18 23:13:11.841100 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:11.841071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:20.840800 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:20.840711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:13:20.842295 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:20.842264    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:13:22.877206 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:22.877174    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:13:22.877610 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:13:22.877431    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:13:30.841547 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:30.841503    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:13:35.876926 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:35.876892    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:13:35.877386 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:13:35.877132    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:13:40.841263 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:40.841219    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:13:49.876717 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:49.876685    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:13:49.877120 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:13:49.876857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:13:50.841491 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:13:50.841443    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:14:00.841645 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:00.841592    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:14:02.876996 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:02.876965    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:14:02.877369 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:14:02.877138    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:14:10.841696 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:10.841648    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:14:14.877094 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:14.877062    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:14:14.877502 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:14:14.877236    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:14:20.841451 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:20.841385    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 18 23:14:26.876878 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:26.876849    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:14:26.877263 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:14:26.876991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:14:30.851055 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:30.851022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:14:30.858860 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:30.858835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:14:38.876668 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:38.876637    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:14:38.877171 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:14:38.876897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:14:52.876725 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:52.876625    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:14:52.877299 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:14:52.876881    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:14:57.256052 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:57.256020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-66f88c7f48-jch2d_f0889743-6d13-4c7f-be6c-e9976a928fe1/main/0.log"
Sep 18 23:14:58.235847 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:14:58.235815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:15:03.876635 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:03.876606    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:15:03.876974 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:03.876762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:13.258426 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.258367    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" containerID="cri-o://05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218" gracePeriod=45
Sep 18 23:15:13.917674 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.917651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:15:13.965425 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965425 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965586 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965426    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965586 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhrrf\" (UniqueName: \"kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965586 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965562    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965724 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965724 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs\") pod \"f0889743-6d13-4c7f-be6c-e9976a928fe1\" (UID: \"f0889743-6d13-4c7f-be6c-e9976a928fe1\") "
Sep 18 23:15:13.965833 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:15:13.965893 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.965801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home" (OuterVolumeSpecName: "home") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:15:13.966140 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.966118    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:13.966290 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.966275    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:13.968000 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.967968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf" (OuterVolumeSpecName: "kube-api-access-dhrrf") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "kube-api-access-dhrrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:15:13.968099 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.968020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:15:13.968099 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.968042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm" (OuterVolumeSpecName: "dshm") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:15:13.974581 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.974561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434" (OuterVolumeSpecName: "kserve-pvc-source") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "pvc-56b90f26-d636-4f5c-84b8-a352543a0b05". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 23:15:13.977078 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:13.977052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0889743-6d13-4c7f-be6c-e9976a928fe1" (UID: "f0889743-6d13-4c7f-be6c-e9976a928fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:15:14.067417 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.067374    2563 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") on node \"ip-10-0-141-13.ec2.internal\" "
Sep 18 23:15:14.067512 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.067420    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:14.067512 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.067436    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0889743-6d13-4c7f-be6c-e9976a928fe1-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:14.067512 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.067448    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0889743-6d13-4c7f-be6c-e9976a928fe1-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:14.067512 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.067460    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhrrf\" (UniqueName: \"kubernetes.io/projected/f0889743-6d13-4c7f-be6c-e9976a928fe1-kube-api-access-dhrrf\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:15:14.094002 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.093987    2563 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-56b90f26-d636-4f5c-84b8-a352543a0b05" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434") on node "ip-10-0-141-13.ec2.internal"
Sep 18 23:15:14.168142 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.168119    2563 reconciler_common.go:299] "Volume detached for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"csi-6fc887ba541289b651f973da1d0d25c46b17eaa86304eabb6cbb830ba08f7559\""
Sep 18 23:15:14.292226 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.292173    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerID="05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218" exitCode=0
Sep 18 23:15:14.292226 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.292219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"
Sep 18 23:15:14.292590 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.292233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" event={"ID":"f0889743-6d13-4c7f-be6c-e9976a928fe1","Type":"ContainerDied","Data":"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"}
Sep 18 23:15:14.292590 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.292272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" event={"ID":"f0889743-6d13-4c7f-be6c-e9976a928fe1","Type":"ContainerDied","Data":"77d9cfac9cc13495e95a73827eafca06d82d36c3bf18fa11ddb7faf8cfa2e9bc"}
Sep 18 23:15:14.292590 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.292293    2563 scope.go:117] "RemoveContainer" containerID="05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"
Sep 18 23:15:14.300758 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.300740    2563 scope.go:117] "RemoveContainer" containerID="05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"
Sep 18 23:15:14.301040 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:14.301020    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218\": container with ID starting with 05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218 not found: ID does not exist" containerID="05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"
Sep 18 23:15:14.301108 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.301051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218"} err="failed to get container status \"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218\": rpc error: code = NotFound desc = could not find container \"05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218\": container with ID starting with 05eda09d8828e15c8dd2005f02f1cc5e0ac9b4c948446d39e4accbacc43d7218 not found: ID does not exist"
Sep 18 23:15:14.315365 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.315340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:15:14.318818 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.318797    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d"]
Sep 18 23:15:14.851890 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:14.851854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-66f88c7f48-jch2d" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 23:15:15.882010 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:15.881978    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" path="/var/lib/kubelet/pods/f0889743-6d13-4c7f-be6c-e9976a928fe1/volumes"
Sep 18 23:15:18.877241 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:18.877211    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:15:18.877640 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:18.877379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:31.878905 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:31.878874    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:15:32.372613 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.372584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/18.log"
Sep 18 23:15:32.373053 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.373036    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/17.log"
Sep 18 23:15:32.373122 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.373071    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523" exitCode=1
Sep 18 23:15:32.373160 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.373143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"}
Sep 18 23:15:32.373207 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.373197    2563 scope.go:117] "RemoveContainer" containerID="4562ff0a55dbccbdc3c56b2aa7684ea374c2769380e3ed7cf3587785f8af4aee"
Sep 18 23:15:32.373484 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:32.373466    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:15:32.373694 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:32.373675    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:33.378893 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:33.378862    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/18.log"
Sep 18 23:15:34.302309 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:34.302275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:15:34.302642 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:34.302624    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:15:34.302820 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:34.302801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:37.982843 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:37.982809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:15:37.983198 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:37.983119    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:15:37.983282 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:37.983262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:43.811707 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.811674    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:15:43.812195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.812004    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main"
Sep 18 23:15:43.812195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.812018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main"
Sep 18 23:15:43.812195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.812092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0889743-6d13-4c7f-be6c-e9976a928fe1" containerName="main"
Sep 18 23:15:43.814876 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.814856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.818325 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.818297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 23:15:43.818325 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.818308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8hsv8\""
Sep 18 23:15:43.818538 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.818305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 23:15:43.818538 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.818379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 23:15:43.825853 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.825829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:15:43.890497 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.890497 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.890706 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.890706 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dx8\" (UniqueName: \"kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.890706 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.890706 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.890665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991662 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991662 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991880 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991880 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dx8\" (UniqueName: \"kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991880 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.991880 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.991743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.992104 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.992018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.992104 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.992083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.992279 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.992257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.994249 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.994230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.994349 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.994334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:43.999362 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:43.999339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dx8\" (UniqueName: \"kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4c8t7\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:44.058597 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.058564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:15:44.062331 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.062280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.064759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.064741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-xflg7\""
Sep 18 23:15:44.072599 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.072575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:15:44.127221 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.127196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:44.193036 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.192991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zwq\" (UniqueName: \"kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.193227 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.193064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.293925 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.293890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.294102 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.294057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zwq\" (UniqueName: \"kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.296448 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.296422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.303049 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.303025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zwq\" (UniqueName: \"kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq\") pod \"rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.374603 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.374573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:44.403591 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.403530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:15:44.405804 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:15:44.405779    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a30afba_503b_4365_9ad7_3909b23bcdf4.slice/crio-8111cfd540b886c1a06e8e271ffcb4f5e5ba1fa51ec1c6c8ff2c641ca547271b WatchSource:0}: Error finding container 8111cfd540b886c1a06e8e271ffcb4f5e5ba1fa51ec1c6c8ff2c641ca547271b: Status 404 returned error can't find the container with id 8111cfd540b886c1a06e8e271ffcb4f5e5ba1fa51ec1c6c8ff2c641ca547271b
Sep 18 23:15:44.433261 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.433166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" event={"ID":"6a30afba-503b-4365-9ad7-3909b23bcdf4","Type":"ContainerStarted","Data":"8111cfd540b886c1a06e8e271ffcb4f5e5ba1fa51ec1c6c8ff2c641ca547271b"}
Sep 18 23:15:44.649027 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:44.648946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:15:44.651861 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:15:44.651832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf673e3e_b678_415c_833f_d66271d57de3.slice/crio-7848f58021dbd2ed65a16f9705142ed5a11ab5412d515327c42341dad6eb51f1 WatchSource:0}: Error finding container 7848f58021dbd2ed65a16f9705142ed5a11ab5412d515327c42341dad6eb51f1: Status 404 returned error can't find the container with id 7848f58021dbd2ed65a16f9705142ed5a11ab5412d515327c42341dad6eb51f1
Sep 18 23:15:45.440231 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.440192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" event={"ID":"6a30afba-503b-4365-9ad7-3909b23bcdf4","Type":"ContainerStarted","Data":"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"}
Sep 18 23:15:45.441926 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.441892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" event={"ID":"df673e3e-b678-415c-833f-d66271d57de3","Type":"ContainerStarted","Data":"7a7feae46fb8616b9a5757b020ccf59ea9f4298524eab53f8c6a4cf45f6d72bc"}
Sep 18 23:15:45.442058 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.441932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" event={"ID":"df673e3e-b678-415c-833f-d66271d57de3","Type":"ContainerStarted","Data":"7848f58021dbd2ed65a16f9705142ed5a11ab5412d515327c42341dad6eb51f1"}
Sep 18 23:15:45.442058 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.441965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:15:45.462469 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.462425    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" podStartSLOduration=2.462390293 podStartE2EDuration="2.462390293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:15:45.460640581 +0000 UTC m=+5085.659348881" watchObservedRunningTime="2026-09-18 23:15:45.462390293 +0000 UTC m=+5085.661098592"
Sep 18 23:15:45.476996 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:45.476946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podStartSLOduration=1.476929961 podStartE2EDuration="1.476929961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:15:45.47598858 +0000 UTC m=+5085.674696880" watchObservedRunningTime="2026-09-18 23:15:45.476929961 +0000 UTC m=+5085.675638263"
Sep 18 23:15:46.128145 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:46.128112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:48.877148 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:48.877045    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:15:48.891408 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:15:48.877237    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:15:54.127618 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:54.127584    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:54.139663 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:54.139640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:15:54.499882 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:15:54.499802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:16:00.877369 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:00.877336    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:16:00.878013 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:00.877575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:16:02.250739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:02.250707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/18.log"
Sep 18 23:16:02.262479 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:02.262451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/18.log"
Sep 18 23:16:02.263273 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:02.263251    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:16:02.273294 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:02.273274    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:16:11.878465 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:11.878437    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:16:11.878856 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:11.878611    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:16:16.449127 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:16.449054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:16:21.575100 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.575063    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:16:21.582236 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.582209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:21.592755    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:  > containerID="d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1" cmd=["/bin/sleep","15"]
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.592834    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" message="" err=<
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:  >
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:21.592852    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerName="main" containerID="cri-o://d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"
Sep 18 23:16:21.593464 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.592916    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerName="main" containerID="cri-o://d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1" gracePeriod=60
Sep 18 23:16:21.943241 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.943219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:16:21.992088 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992237 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992237 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992150    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5dx8\" (UniqueName: \"kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992237 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992179    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992237 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992237 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir\") pod \"6a30afba-503b-4365-9ad7-3909b23bcdf4\" (UID: \"6a30afba-503b-4365-9ad7-3909b23bcdf4\") "
Sep 18 23:16:21.992565 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:21.992565 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home" (OuterVolumeSpecName: "home") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:21.992565 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992546    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-model-cache\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:21.992744 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.992609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:21.994463 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.994436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm" (OuterVolumeSpecName: "dshm") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:21.994570 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.994490    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:16:21.994617 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:21.994567    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8" (OuterVolumeSpecName: "kube-api-access-q5dx8") pod "6a30afba-503b-4365-9ad7-3909b23bcdf4" (UID: "6a30afba-503b-4365-9ad7-3909b23bcdf4"). InnerVolumeSpecName "kube-api-access-q5dx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:16:22.093327 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.093300    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-dshm\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.093327 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.093324    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-home\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.093500 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.093350    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a30afba-503b-4365-9ad7-3909b23bcdf4-tmp-dir\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.093500 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.093358    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a30afba-503b-4365-9ad7-3909b23bcdf4-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.093500 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.093368    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dx8\" (UniqueName: \"kubernetes.io/projected/6a30afba-503b-4365-9ad7-3909b23bcdf4-kube-api-access-q5dx8\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.621353 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.621321    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerID="d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1" exitCode=0
Sep 18 23:16:22.621835 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.621366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" event={"ID":"6a30afba-503b-4365-9ad7-3909b23bcdf4","Type":"ContainerDied","Data":"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"}
Sep 18 23:16:22.621835 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.621369    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"
Sep 18 23:16:22.621835 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.621437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7" event={"ID":"6a30afba-503b-4365-9ad7-3909b23bcdf4","Type":"ContainerDied","Data":"8111cfd540b886c1a06e8e271ffcb4f5e5ba1fa51ec1c6c8ff2c641ca547271b"}
Sep 18 23:16:22.621835 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.621472    2563 scope.go:117] "RemoveContainer" containerID="d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"
Sep 18 23:16:22.631266 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.631249    2563 scope.go:117] "RemoveContainer" containerID="d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"
Sep 18 23:16:22.631601 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:22.631579    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1\": container with ID starting with d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1 not found: ID does not exist" containerID="d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"
Sep 18 23:16:22.631661 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.631611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1"} err="failed to get container status \"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1\": rpc error: code = NotFound desc = could not find container \"d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1\": container with ID starting with d3ae38aff9ef092dedc987aeb95dbf306efee1407ba01bea5fbdbf78a5139eb1 not found: ID does not exist"
Sep 18 23:16:22.642910 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.642888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:16:22.649365 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:22.649340    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4c8t7"]
Sep 18 23:16:23.881780 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:23.881750    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" path="/var/lib/kubelet/pods/6a30afba-503b-4365-9ad7-3909b23bcdf4/volumes"
Sep 18 23:16:25.877045 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:25.877006    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:16:25.877462 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:25.877177    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:16:26.448833 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:26.448796    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:16:36.448668 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:36.448626    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:16:36.604942 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:36.604904    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" containerID="cri-o://7a7feae46fb8616b9a5757b020ccf59ea9f4298524eab53f8c6a4cf45f6d72bc" gracePeriod=45
Sep 18 23:16:38.876674 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:38.876643    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:16:38.877039 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:38.876803    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:16:46.448178 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:46.448138    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:16:46.448600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:46.448240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:16:52.876259 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:52.876227    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:16:52.876736 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:16:52.876426    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:16:56.449139 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:16:56.449096    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:17:03.876947 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:03.876913    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:17:03.877365 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:17:03.877134    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:17:06.448528 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:06.448483    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:17:06.820202 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:06.820171    2563 generic.go:356] "Generic (PLEG): container finished" podID="df673e3e-b678-415c-833f-d66271d57de3" containerID="7a7feae46fb8616b9a5757b020ccf59ea9f4298524eab53f8c6a4cf45f6d72bc" exitCode=0
Sep 18 23:17:06.820347 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:06.820238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" event={"ID":"df673e3e-b678-415c-833f-d66271d57de3","Type":"ContainerDied","Data":"7a7feae46fb8616b9a5757b020ccf59ea9f4298524eab53f8c6a4cf45f6d72bc"}
Sep 18 23:17:06.855167 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:06.855142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:17:07.017334 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.017271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77zwq\" (UniqueName: \"kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq\") pod \"df673e3e-b678-415c-833f-d66271d57de3\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") "
Sep 18 23:17:07.017334 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.017329    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs\") pod \"df673e3e-b678-415c-833f-d66271d57de3\" (UID: \"df673e3e-b678-415c-833f-d66271d57de3\") "
Sep 18 23:17:07.019366 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.019338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq" (OuterVolumeSpecName: "kube-api-access-77zwq") pod "df673e3e-b678-415c-833f-d66271d57de3" (UID: "df673e3e-b678-415c-833f-d66271d57de3"). InnerVolumeSpecName "kube-api-access-77zwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:17:07.019366 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.019346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df673e3e-b678-415c-833f-d66271d57de3" (UID: "df673e3e-b678-415c-833f-d66271d57de3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:17:07.118675 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.118649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zwq\" (UniqueName: \"kubernetes.io/projected/df673e3e-b678-415c-833f-d66271d57de3-kube-api-access-77zwq\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:17:07.118675 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.118671    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df673e3e-b678-415c-833f-d66271d57de3-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:17:07.826137 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.826106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"
Sep 18 23:17:07.826585 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.826098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx" event={"ID":"df673e3e-b678-415c-833f-d66271d57de3","Type":"ContainerDied","Data":"7848f58021dbd2ed65a16f9705142ed5a11ab5412d515327c42341dad6eb51f1"}
Sep 18 23:17:07.826585 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.826241    2563 scope.go:117] "RemoveContainer" containerID="7a7feae46fb8616b9a5757b020ccf59ea9f4298524eab53f8c6a4cf45f6d72bc"
Sep 18 23:17:07.850469 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.850444    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:17:07.853374 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.853351    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-765bf9bb86-9fclx"]
Sep 18 23:17:07.881828 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:07.881805    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df673e3e-b678-415c-833f-d66271d57de3" path="/var/lib/kubelet/pods/df673e3e-b678-415c-833f-d66271d57de3/volumes"
Sep 18 23:17:17.876833 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:17.876804    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:17:17.877210 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:17:17.876969    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:17:28.876711 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:28.876678    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:17:28.877157 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:17:28.876850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:17:42.876983 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:42.876950    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:17:42.877382 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:17:42.877128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:17:56.877368 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:17:56.877291    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:17:56.877762 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:17:56.877480    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:18:07.882267 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:18:07.882238    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:18:07.882736 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:18:07.882447    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:18:22.877051 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:18:22.877020    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:18:22.877477 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:18:22.877193    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:18:36.876343 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:18:36.876307    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:18:36.876933 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:18:36.876565    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:18:49.877276 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:18:49.877245    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:18:49.877676 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:18:49.877433    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:19:03.876690 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:19:03.876653    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:19:03.877108 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:19:03.876814    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:19:14.876798 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:19:14.876712    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:19:14.877306 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:19:14.876928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:19:26.876970 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:19:26.876933    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:19:26.877375 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:19:26.877101    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:19:39.877244 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:19:39.877207    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:19:39.877671 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:19:39.877462    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:19:50.876794 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:19:50.876760    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:19:50.877253 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:19:50.876924    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:05.877162 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:05.877129    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:20:05.877607 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:05.877316    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:20.877083 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:20.877050    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:20:20.877518 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:20.877207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:32.877152 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:32.877119    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:20:33.732358 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.732329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/19.log"
Sep 18 23:20:33.732888 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.732866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/18.log"
Sep 18 23:20:33.732944 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.732916    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c" exitCode=1
Sep 18 23:20:33.733014 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.732990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"}
Sep 18 23:20:33.733065 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.733054    2563 scope.go:117] "RemoveContainer" containerID="4190ae9d2a8750a8b8bb7d195b5009944af3c15ab18d1062684e397482eb9523"
Sep 18 23:20:33.733463 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:33.733439    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:20:33.733661 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:33.733638    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:34.301858 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:34.301794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:20:34.738601 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:34.738571    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/19.log"
Sep 18 23:20:34.738903 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:34.738885    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:20:34.739070 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:34.739050    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:37.983613 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:37.983582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:20:37.983989 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:37.983895    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:20:37.984065 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:37.984046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:20:52.876346 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:20:52.876314    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:20:52.878837 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:20:52.876516    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:21:02.278597 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:02.278570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/19.log"
Sep 18 23:21:02.287098 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:02.287076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/19.log"
Sep 18 23:21:02.289226 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:02.289206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:21:02.297962 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:02.297943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:21:04.877318 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:04.877286    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:21:04.877825 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:21:04.877533    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:21:15.877226 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:15.877197    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:21:15.877623 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:21:15.877358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:21:30.877257 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:30.877227    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:21:30.877795 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:21:30.877456    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:21:45.877128 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:21:45.877096    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:21:45.877641 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:21:45.877271    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:22:00.877044 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:22:00.877005    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:22:00.877531 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:22:00.877243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:22:15.876778 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:22:15.876695    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:22:15.877231 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:22:15.876872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:22:26.876772 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:22:26.876743    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:22:26.877226 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:22:26.876914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:22:41.878310 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:22:41.878280    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:22:41.878731 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:22:41.878464    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:22:55.877174 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:22:55.877141    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:22:55.877609 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:22:55.877311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:23:08.876899 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:23:08.876870    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:23:08.877341 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:23:08.877044    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:23:19.876527 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:23:19.876490    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:23:19.876976 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:23:19.876684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:23:33.876692 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:23:33.876662    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:23:33.877167 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:23:33.876842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:23:48.877128 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:23:48.877054    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:23:48.877615 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:23:48.877212    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:24:03.876976 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:24:03.876943    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:24:03.877491 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:24:03.877111    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:24:15.877241 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:24:15.877211    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:24:15.877836 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:24:15.877359    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:24:30.876482 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:24:30.876451    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:24:30.876867 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:24:30.876644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:24:42.877204 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:24:42.877173    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:24:42.877624 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:24:42.877334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:24:55.877065 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:24:55.877030    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:24:55.877574 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:24:55.877285    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:10.876595 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:10.876549    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:25:10.877053 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:10.876732    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:21.879249 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:21.879174    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:25:21.879723 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:21.879367    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:35.877008 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:35.876978    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:25:36.040818 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.040796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/20.log"
Sep 18 23:25:36.041224 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.041207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/19.log"
Sep 18 23:25:36.041324 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.041245    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567" exitCode=1
Sep 18 23:25:36.041378 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.041315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"}
Sep 18 23:25:36.041378 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.041370    2563 scope.go:117] "RemoveContainer" containerID="3b8d8fdfbf2a66f7ae31d967693d03c9696b5738c5595eeae0176ad6d39e7c0c"
Sep 18 23:25:36.041692 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:36.041666    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:25:36.041918 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:36.041891    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:37.049754 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:37.049728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/20.log"
Sep 18 23:25:37.983542 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:37.983513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:25:37.983822 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:37.983810    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:25:37.983974 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:37.983958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:44.301814 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:44.301779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:25:44.302260 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:44.302123    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:25:44.302303 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:44.302277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:25:56.876778 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:25:56.876742    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:25:56.879027 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:25:56.876897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:26:02.301467 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:02.301440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/20.log"
Sep 18 23:26:02.312589 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:02.312561    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/20.log"
Sep 18 23:26:02.313852 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:02.313833    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:26:02.323707 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:02.323687    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:26:09.876983 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:09.876953    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:26:09.877365 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:26:09.877121    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:26:20.876930 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:20.876899    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:26:20.877325 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:26:20.877083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:26:33.876981 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:33.876945    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:26:33.877498 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:26:33.877162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:26:46.876570 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:46.876488    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:26:46.876915 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:26:46.876665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:26:58.876745 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:26:58.876719    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:26:58.877091 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:26:58.876872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:27:11.878542 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:11.878508    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:27:11.879023 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:27:11.878680    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:27:14.799963 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:14.799933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh_5794c3e9-efb3-451c-9257-94778463c017/main/0.log"
Sep 18 23:27:16.031072 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:16.031035    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:27:17.810437 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:17.810386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:27:23.877214 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:23.877182    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:27:23.877705 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:27:23.877341    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:27:27.810712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:27.810671    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:27:31.050232 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:31.050169    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" containerID="cri-o://247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd" gracePeriod=45
Sep 18 23:27:36.876789 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:36.876757    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:27:36.877182 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:27:36.876940    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:27:37.810547 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:37.810504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:27:37.810708 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:37.810599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:27:47.809846 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:47.809802    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:27:50.876627 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:50.876595    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:27:50.877003 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:27:50.876751    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:27:57.810460 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:27:57.810416    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:28:01.289222 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.289198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:28:01.360101 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.360074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs\") pod \"5794c3e9-efb3-451c-9257-94778463c017\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") "
Sep 18 23:28:01.360214 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.360114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlht4\" (UniqueName: \"kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4\") pod \"5794c3e9-efb3-451c-9257-94778463c017\" (UID: \"5794c3e9-efb3-451c-9257-94778463c017\") "
Sep 18 23:28:01.362186 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.362158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5794c3e9-efb3-451c-9257-94778463c017" (UID: "5794c3e9-efb3-451c-9257-94778463c017"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:28:01.362278 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.362226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4" (OuterVolumeSpecName: "kube-api-access-vlht4") pod "5794c3e9-efb3-451c-9257-94778463c017" (UID: "5794c3e9-efb3-451c-9257-94778463c017"). InnerVolumeSpecName "kube-api-access-vlht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:28:01.461464 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.461440    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5794c3e9-efb3-451c-9257-94778463c017-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:28:01.461464 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.461462    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlht4\" (UniqueName: \"kubernetes.io/projected/5794c3e9-efb3-451c-9257-94778463c017-kube-api-access-vlht4\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:28:01.702239 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.702168    2563 generic.go:356] "Generic (PLEG): container finished" podID="5794c3e9-efb3-451c-9257-94778463c017" containerID="247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd" exitCode=0
Sep 18 23:28:01.702239 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.702216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" event={"ID":"5794c3e9-efb3-451c-9257-94778463c017","Type":"ContainerDied","Data":"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"}
Sep 18 23:28:01.702384 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.702253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh" event={"ID":"5794c3e9-efb3-451c-9257-94778463c017","Type":"ContainerDied","Data":"2266b8954a50f91de3ca31d06250d9fbaa73b8c4c4ae7ba15a47eb49c9d2e6f7"}
Sep 18 23:28:01.702384 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.702275    2563 scope.go:117] "RemoveContainer" containerID="247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"
Sep 18 23:28:01.702384 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.702216    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"
Sep 18 23:28:01.711633 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.711616    2563 scope.go:117] "RemoveContainer" containerID="247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"
Sep 18 23:28:01.711896 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:01.711877    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd\": container with ID starting with 247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd not found: ID does not exist" containerID="247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"
Sep 18 23:28:01.711964 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.711905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd"} err="failed to get container status \"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd\": rpc error: code = NotFound desc = could not find container \"247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd\": container with ID starting with 247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd not found: ID does not exist"
Sep 18 23:28:01.723814 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.723788    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:28:01.727831 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.727810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh"]
Sep 18 23:28:01.882370 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:01.882342    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5794c3e9-efb3-451c-9257-94778463c017" path="/var/lib/kubelet/pods/5794c3e9-efb3-451c-9257-94778463c017/volumes"
Sep 18 23:28:02.876528 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:02.876483    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:28:02.877107 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:02.876751    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:28:14.877227 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:14.877152    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:28:14.877640 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:14.877345    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:28:25.877251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:25.877219    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:28:25.877630 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:25.877418    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:28:31.632987 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.632952    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633304    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633313    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633326    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerName="main"
Sep 18 23:28:31.633382 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633380    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a30afba-503b-4365-9ad7-3909b23bcdf4" containerName="main"
Sep 18 23:28:31.633640 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633388    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df673e3e-b678-415c-833f-d66271d57de3" containerName="main"
Sep 18 23:28:31.633640 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.633408    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5794c3e9-efb3-451c-9257-94778463c017" containerName="main"
Sep 18 23:28:31.636271 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.636249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.639745 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.639722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-tmhsq\""
Sep 18 23:28:31.639900 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.639729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 23:28:31.640058 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.640013    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 23:28:31.640058 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.640041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 23:28:31.640205 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.640098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-zwj7d\""
Sep 18 23:28:31.645521 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.645496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:28:31.685014 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.684982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhk46\" (UniqueName: \"kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.685163 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.685056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.785632 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.785591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhk46\" (UniqueName: \"kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.785807 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.785654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.788329 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.788303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.793538 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.793516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhk46\" (UniqueName: \"kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46\") pod \"prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:31.947811 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:31.947736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:32.220752 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:32.220721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:28:32.223352 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:28:32.223325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd8f2f_5550_41bc_8184_b1518639868b.slice/crio-6ff7d9c827be6f898fb94dddf73fa4fd72dc5c895ceda38c2352a451ea25a338 WatchSource:0}: Error finding container 6ff7d9c827be6f898fb94dddf73fa4fd72dc5c895ceda38c2352a451ea25a338: Status 404 returned error can't find the container with id 6ff7d9c827be6f898fb94dddf73fa4fd72dc5c895ceda38c2352a451ea25a338
Sep 18 23:28:32.846732 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:32.846691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" event={"ID":"27fd8f2f-5550-41bc-8184-b1518639868b","Type":"ContainerStarted","Data":"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"}
Sep 18 23:28:32.847254 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:32.846760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" event={"ID":"27fd8f2f-5550-41bc-8184-b1518639868b","Type":"ContainerStarted","Data":"6ff7d9c827be6f898fb94dddf73fa4fd72dc5c895ceda38c2352a451ea25a338"}
Sep 18 23:28:32.847589 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:32.847561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:28:40.876952 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:40.876920    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:28:40.877385 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:40.877085    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:28:52.876737 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:28:52.876704    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:28:52.877190 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:28:52.876862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:29:04.858596 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:04.858566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:29:04.876714 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:04.876667    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podStartSLOduration=33.876651181 podStartE2EDuration="33.876651181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:28:32.866273498 +0000 UTC m=+5853.064981798" watchObservedRunningTime="2026-09-18 23:29:04.876651181 +0000 UTC m=+5885.075359480"
Sep 18 23:29:06.876321 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:06.876291    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:29:06.876764 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:29:06.876505    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:29:18.876895 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:18.876859    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:29:18.877326 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:29:18.877037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:29:33.877240 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:33.877204    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:29:33.877864 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:29:33.877417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:29:46.876365 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:46.876288    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:29:46.876772 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:29:46.876536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:29:58.877195 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:29:58.877166    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:29:58.877660 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:29:58.877330    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:10.876990 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:10.876955    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:30:10.877388 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:10.877122    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:22.876989 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:22.876957    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:30:22.877421 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:22.877130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:36.876816 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:36.876785    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:30:37.381890 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.381866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/21.log"
Sep 18 23:30:37.382277 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.382257    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/20.log"
Sep 18 23:30:37.382389 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.382298    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922" exitCode=1
Sep 18 23:30:37.382389 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.382331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" event={"ID":"b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa","Type":"ContainerDied","Data":"f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"}
Sep 18 23:30:37.382494 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.382406    2563 scope.go:117] "RemoveContainer" containerID="0e1fb3e2d4a05a826ebcf6afeedda68a98e64e7e33a7dd5623dbb4a781e22567"
Sep 18 23:30:37.382697 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.382680    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:30:37.382880 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:37.382861    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:37.983368 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:37.983341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:30:38.387984 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:38.387958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/21.log"
Sep 18 23:30:38.388272 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:38.388254    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:30:38.388474 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:38.388452    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:44.302443 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:44.302406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp"
Sep 18 23:30:44.302938 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:44.302804    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:30:44.303041 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:44.303012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:55.877230 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:55.877192    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:30:55.877646 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:30:55.877445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:30:58.143477 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:30:58.143442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:31:02.326831 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:02.326806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/21.log"
Sep 18 23:31:02.338448 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:02.338425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:31:02.338914 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:02.338896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa/main/21.log"
Sep 18 23:31:02.349298 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:02.349283    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dvlk7_91576d05-55c7-41e0-925e-1d2573f17265/ovn-acl-logging/0.log"
Sep 18 23:31:04.857907 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:04.857868    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:31:07.876818 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:07.876785    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:31:07.877303 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:31:07.876944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:31:13.165739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:13.165675    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main" containerID="cri-o://3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb" gracePeriod=45
Sep 18 23:31:14.858518 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:14.858432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:31:19.877365 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:19.877332    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:31:19.877776 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:31:19.877523    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:31:24.858708 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:24.858664    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:31:24.859082 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:24.858772    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:31:34.858091 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:34.858052    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 23:31:34.876477 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:34.876450    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:31:34.876640 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:31:34.876622    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:31:43.417300 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.417275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:31:43.505519 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.505442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs\") pod \"27fd8f2f-5550-41bc-8184-b1518639868b\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") "
Sep 18 23:31:43.505519 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.505491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhk46\" (UniqueName: \"kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46\") pod \"27fd8f2f-5550-41bc-8184-b1518639868b\" (UID: \"27fd8f2f-5550-41bc-8184-b1518639868b\") "
Sep 18 23:31:43.507653 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.507621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27fd8f2f-5550-41bc-8184-b1518639868b" (UID: "27fd8f2f-5550-41bc-8184-b1518639868b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:31:43.507759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.507693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46" (OuterVolumeSpecName: "kube-api-access-lhk46") pod "27fd8f2f-5550-41bc-8184-b1518639868b" (UID: "27fd8f2f-5550-41bc-8184-b1518639868b"). InnerVolumeSpecName "kube-api-access-lhk46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:31:43.606158 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.606123    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27fd8f2f-5550-41bc-8184-b1518639868b-tls-certs\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:31:43.606158 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.606155    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhk46\" (UniqueName: \"kubernetes.io/projected/27fd8f2f-5550-41bc-8184-b1518639868b-kube-api-access-lhk46\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:31:43.677433 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.677384    2563 generic.go:356] "Generic (PLEG): container finished" podID="27fd8f2f-5550-41bc-8184-b1518639868b" containerID="3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb" exitCode=0
Sep 18 23:31:43.677546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.677447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" event={"ID":"27fd8f2f-5550-41bc-8184-b1518639868b","Type":"ContainerDied","Data":"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"}
Sep 18 23:31:43.677546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.677451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"
Sep 18 23:31:43.677546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.677488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6" event={"ID":"27fd8f2f-5550-41bc-8184-b1518639868b","Type":"ContainerDied","Data":"6ff7d9c827be6f898fb94dddf73fa4fd72dc5c895ceda38c2352a451ea25a338"}
Sep 18 23:31:43.677546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.677512    2563 scope.go:117] "RemoveContainer" containerID="3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"
Sep 18 23:31:43.686559 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.686540    2563 scope.go:117] "RemoveContainer" containerID="3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"
Sep 18 23:31:43.686836 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:31:43.686815    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb\": container with ID starting with 3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb not found: ID does not exist" containerID="3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"
Sep 18 23:31:43.686938 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.686840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb"} err="failed to get container status \"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb\": rpc error: code = NotFound desc = could not find container \"3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb\": container with ID starting with 3397bc3a117f353bb77ee1bc97cd08092fa2a39fb49adf015486a7126af21cbb not found: ID does not exist"
Sep 18 23:31:43.700143 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.700117    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:31:43.703342 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.703319    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-65d549b76d-bnhq6"]
Sep 18 23:31:43.881794 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:43.881769    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" path="/var/lib/kubelet/pods/27fd8f2f-5550-41bc-8184-b1518639868b/volumes"
Sep 18 23:31:49.877391 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:31:49.877363    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:31:49.877779 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:31:49.877556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:32:01.878334 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:32:01.878295    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:32:01.878881 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:32:01.878530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:32:15.876372 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:32:15.876339    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:32:15.876986 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:32:15.876548    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:32:28.876453 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:32:28.876418    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:32:28.876883 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:32:28.876646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:32:42.877252 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:32:42.877222    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:32:42.877712 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:32:42.877378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:32:57.877170 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:32:57.877083    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:32:57.877619 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:32:57.877290    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:08.877346 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:08.877317    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:33:08.877731 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:33:08.877488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:10.979623 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.979582    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wgt88/must-gather-79nv5"]
Sep 18 23:33:10.980090 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.980008    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main"
Sep 18 23:33:10.980090 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.980027    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main"
Sep 18 23:33:10.980219 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.980134    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fd8f2f-5550-41bc-8184-b1518639868b" containerName="main"
Sep 18 23:33:10.983109 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.983085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:10.985497 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.985472    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wgt88\"/\"kube-root-ca.crt\""
Sep 18 23:33:10.985618 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:10.985473    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wgt88\"/\"openshift-service-ca.crt\""
Sep 18 23:33:11.000347 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.000311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wgt88/must-gather-79nv5"]
Sep 18 23:33:11.027557 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.027530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.027683 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.027581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfmc\" (UniqueName: \"kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.128445 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.128411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.128593 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.128505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfmc\" (UniqueName: \"kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.128693 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.128673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.136656 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.136637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfmc\" (UniqueName: \"kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc\") pod \"must-gather-79nv5\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") " pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.299680 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.299598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:33:11.577812 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.577780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wgt88/must-gather-79nv5"]
Sep 18 23:33:11.580537 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:33:11.580509    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0118b7ec_621b_463a_b532_4bb7d4ac202e.slice/crio-0c1d451224af651c640a90996527d43d258eb7deceebd35728c846a4bea74cde WatchSource:0}: Error finding container 0c1d451224af651c640a90996527d43d258eb7deceebd35728c846a4bea74cde: Status 404 returned error can't find the container with id 0c1d451224af651c640a90996527d43d258eb7deceebd35728c846a4bea74cde
Sep 18 23:33:11.582198 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:11.582180    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 23:33:12.069827 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:12.069790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wgt88/must-gather-79nv5" event={"ID":"0118b7ec-621b-463a-b532-4bb7d4ac202e","Type":"ContainerStarted","Data":"0c1d451224af651c640a90996527d43d258eb7deceebd35728c846a4bea74cde"}
Sep 18 23:33:18.104727 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:18.104681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wgt88/must-gather-79nv5" event={"ID":"0118b7ec-621b-463a-b532-4bb7d4ac202e","Type":"ContainerStarted","Data":"642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1"}
Sep 18 23:33:18.105220 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:18.104737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wgt88/must-gather-79nv5" event={"ID":"0118b7ec-621b-463a-b532-4bb7d4ac202e","Type":"ContainerStarted","Data":"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"}
Sep 18 23:33:18.120387 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:18.120330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wgt88/must-gather-79nv5" podStartSLOduration=2.0900345 podStartE2EDuration="8.120309187s" totalImagesPullingTime="6.030274687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:33:18.119348316 +0000 UTC m=+6138.318056615" watchObservedRunningTime="2026-09-18 23:33:18.120309187 +0000 UTC m=+6138.319017488"
Sep 18 23:33:20.877028 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:20.876995    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:33:20.877423 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:33:20.877217    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:33.800271 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:33.800237    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:33:33.876739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:33.876711    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:33:33.876899 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:33:33.876879    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:34.590712 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:34.590679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:33:35.327492 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:35.327453    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:33:36.049574 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:36.049542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:33:36.957681 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:36.957644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:36.972380 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:36.972355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:36.986360 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:36.986341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:37.121595 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:37.121526    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:38.007053 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:38.007014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:38.107299 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:38.107273    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:38.138142 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:38.138121    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:38.289101 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:38.289026    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:39.583607 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:39.583573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:39.649087 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:39.649058    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:39.664319 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:39.664288    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:39.795969 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:39.795940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:40.791207 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:40.791171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:40.912296 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:40.912265    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:40.925843 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:40.925824    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:41.057122 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:41.057044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:42.019604 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:42.019569    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:42.038140 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:42.038110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:42.053307 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:42.053283    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:42.186710 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:42.186680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:43.086099 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:43.086065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:43.108385 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:43.108353    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:43.123446 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:43.123425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:43.252481 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:43.252455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:44.174119 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:44.174088    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:44.190583 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:44.190558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:44.205677 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:44.205655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:44.336162 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:44.336129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:44.876389 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:44.876350    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:33:44.876682 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:33:44.876574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:45.242376 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:45.242291    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:45.260389 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:45.260365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:45.274282 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:45.274259    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:45.410386 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:45.410357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:46.286074 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:46.286039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:46.305125 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:46.305104    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:46.318690 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:46.318670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:46.441968 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:46.441933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:47.419073 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:47.419033    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:47.479547 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:47.479520    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:47.494581 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:47.494558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:47.631180 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:47.631140    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:48.549034 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:48.548999    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:48.569656 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:48.569626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:48.582530 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:48.582503    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:48.711931 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:48.711903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:49.602565 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:49.602530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:49.620011 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:49.619982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:49.633546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:49.633527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:49.762861 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:49.762834    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:50.642650 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:50.642617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:50.663974 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:50.663946    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:50.689750 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:50.689708    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:50.882243 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:50.882215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:51.815611 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:51.815578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:51.833562 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:51.833534    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:51.847136 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:51.847118    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:51.981084 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:51.981053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:52.844722 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:52.844685    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:52.871458 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:52.871429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:52.890655 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:52.890620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:53.012109 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:53.012078    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:53.964339 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:53.964306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:53.983881 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:53.983856    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:53.997869 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:53.997849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:54.126495 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:54.126461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:55.018921 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:55.018887    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:55.039832 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:55.039806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:55.052941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:55.052924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:55.185854 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:55.185825    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:56.058378 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:56.058342    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:56.077614 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:56.077585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:56.097788 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:56.097762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:56.231919 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:56.231882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:56.876739 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:56.876707    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:33:56.876944 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:33:56.876923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:33:57.094413 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:57.094367    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:57.110008 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:57.109980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:57.125646 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:57.125626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:57.260654 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:57.260550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:58.110328 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:58.110291    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:58.179023 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:58.178991    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:58.193784 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:58.193761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:58.323842 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:58.323810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:33:59.194835 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:59.194791    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:33:59.215054 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:59.215028    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:33:59.228748 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:59.228720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:33:59.355378 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:33:59.355348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:00.315452 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:00.315415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:00.335438 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:00.335413    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:00.352484 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:00.352459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:00.488379 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:00.488326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:01.375914 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:01.375878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:01.406166 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:01.406139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:01.420035 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:01.420005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:01.545128 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:01.545086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:02.446117 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:02.446082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:02.476566 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:02.476525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:02.493591 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:02.493567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:02.624919 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:02.624888    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:03.528867 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:03.528832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:03.550727 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:03.550703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:03.564069 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:03.564049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:03.690164 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:03.690134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:04.563246 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:04.563213    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:04.585118 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:04.585087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:04.598943 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:04.598921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:04.729042 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:04.729013    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:05.663628 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:05.663596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:05.680766 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:05.680741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:05.693286 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:05.693262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:05.822149 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:05.822119    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:06.691264 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:06.691225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:06.708957 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:06.708930    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:06.722941 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:06.722922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:06.852100 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:06.852064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:07.726783 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:07.726749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:07.748148 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:07.748122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:07.765345 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:07.765322    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:07.876449 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:07.876421    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:34:07.876669 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:07.876641    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:34:07.913918 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:07.913888    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:08.927964 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:08.927930    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:08.950687 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:08.950661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:08.964717 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:08.964692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:09.095414 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:09.095364    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:10.003390 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:10.003355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:10.023132 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:10.023105    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:10.041232 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:10.041207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:10.173576 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:10.173546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:11.076078 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:11.076040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:11.092315 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:11.092284    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:11.106350 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:11.106325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:11.242439 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:11.242386    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:12.138298 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:12.138264    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:12.163921 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:12.163898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:12.176045 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:12.176025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:12.304313 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:12.304277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:13.166786 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:13.166757    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:13.184407 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:13.184368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:13.198606 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:13.198587    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:13.332205 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:13.332173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:14.182496 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:14.182461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:14.203935 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:14.203904    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:14.220022 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:14.220001    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:14.356415 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:14.356368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:15.208431 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:15.208337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:15.226884 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:15.226855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:15.241073 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:15.241053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:15.369108 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:15.369070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:16.233332 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:16.233294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:16.251153 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:16.251128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:16.264849 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:16.264827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:16.389442 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:16.389379    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:17.432040 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:17.432013    2563 generic.go:356] "Generic (PLEG): container finished" podID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerID="adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a" exitCode=0
Sep 18 23:34:17.432460 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:17.432081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wgt88/must-gather-79nv5" event={"ID":"0118b7ec-621b-463a-b532-4bb7d4ac202e","Type":"ContainerDied","Data":"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"}
Sep 18 23:34:17.432460 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:17.432382    2563 scope.go:117] "RemoveContainer" containerID="adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"
Sep 18 23:34:17.910546 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:17.910513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wgt88_must-gather-79nv5_0118b7ec-621b-463a-b532-4bb7d4ac202e/gather/0.log"
Sep 18 23:34:21.667231 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:21.667204    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sdwt8_53076ba4-5cba-44f1-bf24-28f762dd893d/global-pull-secret-syncer/0.log"
Sep 18 23:34:21.834148 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:21.834122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tpwpt_bbc48f62-0f58-452d-ba39-5b456370065a/konnectivity-agent/0.log"
Sep 18 23:34:21.878787 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:21.878759    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:34:21.878954 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:21.878935    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:34:21.889036 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:21.889017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-13.ec2.internal_0ea39be3926c31923b8b3abafa3a95cc/haproxy/0.log"
Sep 18 23:34:23.391715 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.391680    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-wgt88/must-gather-79nv5"]
Sep 18 23:34:23.392124 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.391851    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-wgt88/must-gather-79nv5" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="copy" containerID="cri-o://642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1" gracePeriod=2
Sep 18 23:34:23.396957 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.396924    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-wgt88/must-gather-79nv5"]
Sep 18 23:34:23.623099 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.623079    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wgt88_must-gather-79nv5_0118b7ec-621b-463a-b532-4bb7d4ac202e/copy/0.log"
Sep 18 23:34:23.625593 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.625575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:34:23.627729 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:23.627703    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-79nv5\" is forbidden: User \"system:node:ip-10-0-141-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wgt88\": no relationship found between node 'ip-10-0-141-13.ec2.internal' and this object" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:34:23.683851 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.683797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnfmc\" (UniqueName: \"kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc\") pod \"0118b7ec-621b-463a-b532-4bb7d4ac202e\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") "
Sep 18 23:34:23.683851 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.683827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output\") pod \"0118b7ec-621b-463a-b532-4bb7d4ac202e\" (UID: \"0118b7ec-621b-463a-b532-4bb7d4ac202e\") "
Sep 18 23:34:23.685918 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.685892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc" (OuterVolumeSpecName: "kube-api-access-hnfmc") pod "0118b7ec-621b-463a-b532-4bb7d4ac202e" (UID: "0118b7ec-621b-463a-b532-4bb7d4ac202e"). InnerVolumeSpecName "kube-api-access-hnfmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:34:23.691495 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.691475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0118b7ec-621b-463a-b532-4bb7d4ac202e" (UID: "0118b7ec-621b-463a-b532-4bb7d4ac202e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:34:23.784256 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.784231    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnfmc\" (UniqueName: \"kubernetes.io/projected/0118b7ec-621b-463a-b532-4bb7d4ac202e-kube-api-access-hnfmc\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:34:23.784256 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.784253    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0118b7ec-621b-463a-b532-4bb7d4ac202e-must-gather-output\") on node \"ip-10-0-141-13.ec2.internal\" DevicePath \"\""
Sep 18 23:34:23.882433 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:23.882391    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" path="/var/lib/kubelet/pods/0118b7ec-621b-463a-b532-4bb7d4ac202e/volumes"
Sep 18 23:34:24.462528 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.462500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wgt88_must-gather-79nv5_0118b7ec-621b-463a-b532-4bb7d4ac202e/copy/0.log"
Sep 18 23:34:24.463017 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.462871    2563 generic.go:356] "Generic (PLEG): container finished" podID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerID="642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1" exitCode=143
Sep 18 23:34:24.463017 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.462917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wgt88/must-gather-79nv5"
Sep 18 23:34:24.463017 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.462975    2563 scope.go:117] "RemoveContainer" containerID="642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1"
Sep 18 23:34:24.470757 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.470738    2563 scope.go:117] "RemoveContainer" containerID="adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"
Sep 18 23:34:24.484009 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.483987    2563 scope.go:117] "RemoveContainer" containerID="642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1"
Sep 18 23:34:24.484278 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:24.484260    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1\": container with ID starting with 642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1 not found: ID does not exist" containerID="642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1"
Sep 18 23:34:24.484318 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.484287    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1"} err="failed to get container status \"642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1\": rpc error: code = NotFound desc = could not find container \"642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1\": container with ID starting with 642907c0fe44fc94286681a7ffb7ce182f9904b847b594ab71de3bed5a75c5a1 not found: ID does not exist"
Sep 18 23:34:24.484318 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.484304    2563 scope.go:117] "RemoveContainer" containerID="adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"
Sep 18 23:34:24.484564 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:24.484545    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a\": container with ID starting with adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a not found: ID does not exist" containerID="adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"
Sep 18 23:34:24.484616 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:24.484570    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a"} err="failed to get container status \"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a\": rpc error: code = NotFound desc = could not find container \"adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a\": container with ID starting with adb96cbf4adbfc200d370f47a140ab6fec064bbe02b967e082558e02c0bde93a not found: ID does not exist"
Sep 18 23:34:27.122610 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:27.122574    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-7df2d_70b36827-60f5-46e3-aefb-c52dd57fbcbb/authorino/0.log"
Sep 18 23:34:27.166759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:27.166733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-4srxn_4384570e-0e13-4d35-9708-32fc77e1bbb1/manager/0.log"
Sep 18 23:34:27.195600 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:27.195576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4sgbx_7c2d9e8c-5c3d-49ad-91fb-de024c657704/manager/0.log"
Sep 18 23:34:27.345820 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:27.345791    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vfzqt_d161bb61-04e8-4fa7-bbeb-15ea3bec964f/manager/0.log"
Sep 18 23:34:29.340668 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:29.340645    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfl9r_781e40d8-b59e-40bb-9de7-8366751c58a1/node-exporter/0.log"
Sep 18 23:34:29.366589 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:29.366562    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfl9r_781e40d8-b59e-40bb-9de7-8366751c58a1/kube-rbac-proxy/0.log"
Sep 18 23:34:29.389888 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:29.389866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfl9r_781e40d8-b59e-40bb-9de7-8366751c58a1/init-textfile/0.log"
Sep 18 23:34:31.458230 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"]
Sep 18 23:34:31.458714 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458640    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="gather"
Sep 18 23:34:31.458714 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="gather"
Sep 18 23:34:31.458714 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="copy"
Sep 18 23:34:31.458714 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="copy"
Sep 18 23:34:31.458926 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458785    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="gather"
Sep 18 23:34:31.458926 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.458797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0118b7ec-621b-463a-b532-4bb7d4ac202e" containerName="copy"
Sep 18 23:34:31.461778 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.461756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.464145 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.464126    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"openshift-service-ca.crt\""
Sep 18 23:34:31.464234 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.464199    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"kube-root-ca.crt\""
Sep 18 23:34:31.464876 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.464858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6w2r\"/\"default-dockercfg-hpdwh\""
Sep 18 23:34:31.468722 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.468563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"]
Sep 18 23:34:31.539649 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.539623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pt8w\" (UniqueName: \"kubernetes.io/projected/4d7400ff-73dc-41e8-8655-d70d6586d58e-kube-api-access-7pt8w\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.539759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.539659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-proc\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.539759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.539675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-lib-modules\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.539759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.539696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-podres\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.539759 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.539741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-sys\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640548 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-podres\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640646 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-sys\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640646 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pt8w\" (UniqueName: \"kubernetes.io/projected/4d7400ff-73dc-41e8-8655-d70d6586d58e-kube-api-access-7pt8w\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640646 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-proc\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640646 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-lib-modules\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640798 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-podres\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640798 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-proc\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640798 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-sys\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.640798 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.640765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d7400ff-73dc-41e8-8655-d70d6586d58e-lib-modules\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.649302 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.649276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pt8w\" (UniqueName: \"kubernetes.io/projected/4d7400ff-73dc-41e8-8655-d70d6586d58e-kube-api-access-7pt8w\") pod \"perf-node-gather-daemonset-gmwfh\" (UID: \"4d7400ff-73dc-41e8-8655-d70d6586d58e\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:31.772437 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:31.772369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:32.045532 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.045507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"]
Sep 18 23:34:32.048136 ip-10-0-141-13 kubenswrapper[2563]: W0918 23:34:32.048105    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d7400ff_73dc_41e8_8655_d70d6586d58e.slice/crio-6dc5c5c7a88bb987dee48ec3bedcef1b6aea7039b284701784cdc088b0da100d WatchSource:0}: Error finding container 6dc5c5c7a88bb987dee48ec3bedcef1b6aea7039b284701784cdc088b0da100d: Status 404 returned error can't find the container with id 6dc5c5c7a88bb987dee48ec3bedcef1b6aea7039b284701784cdc088b0da100d
Sep 18 23:34:32.289240 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.289216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67b6fb4769-2gf85_26cf3522-8185-49ea-bbee-4896e5224e64/console/0.log"
Sep 18 23:34:32.503094 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.503067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh" event={"ID":"4d7400ff-73dc-41e8-8655-d70d6586d58e","Type":"ContainerStarted","Data":"2c0de2a8ffcc34a65e6eeb2d16f5ecd714aef5c84ef77a7981225607a42eb355"}
Sep 18 23:34:32.503431 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.503114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh" event={"ID":"4d7400ff-73dc-41e8-8655-d70d6586d58e","Type":"ContainerStarted","Data":"6dc5c5c7a88bb987dee48ec3bedcef1b6aea7039b284701784cdc088b0da100d"}
Sep 18 23:34:32.503431 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.503144    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:32.521126 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:32.521081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh" podStartSLOduration=1.5210677750000001 podStartE2EDuration="1.521067775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:34:32.520565661 +0000 UTC m=+6212.719273961" watchObservedRunningTime="2026-09-18 23:34:32.521067775 +0000 UTC m=+6212.719776106"
Sep 18 23:34:34.050756 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:34.050728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2l79_9ee153d0-4d3a-432a-a011-cab546f03d5d/dns/0.log"
Sep 18 23:34:34.071251 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:34.071231    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2l79_9ee153d0-4d3a-432a-a011-cab546f03d5d/kube-rbac-proxy/0.log"
Sep 18 23:34:34.184140 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:34.184114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9gnp_02a42efa-8572-4d8a-817a-f8d6542f0f4d/dns-node-resolver/0.log"
Sep 18 23:34:34.693339 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:34.693313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55ff9774c5-tss26_67093347-f081-4c7f-b51f-1e31b4ef3b76/registry/0.log"
Sep 18 23:34:34.768504 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:34.768477    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8w62_7f5e819a-8e56-41f7-ab2a-0d9f8a382813/node-ca/0.log"
Sep 18 23:34:35.845722 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:35.845692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:34:36.453656 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:36.453626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h8xj8_cfeec5cb-1610-4b74-886b-d9c3740c0383/serve-healthcheck-canary/0.log"
Sep 18 23:34:36.877245 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:36.877219    2563 scope.go:117] "RemoveContainer" containerID="f9c6737d3b8fb0b02162bc93ce0fed48fdd2b27c1cc0bd8319c01bd1ef7ad922"
Sep 18 23:34:36.877609 ip-10-0-141-13 kubenswrapper[2563]: E0918 23:34:36.877382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa)\"" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp" podUID="b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa"
Sep 18 23:34:37.125573 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:37.125541    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbczd_05b8f250-4be6-4182-9706-303d1f6ee7b9/kube-rbac-proxy/0.log"
Sep 18 23:34:37.148928 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:37.148864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbczd_05b8f250-4be6-4182-9706-303d1f6ee7b9/exporter/0.log"
Sep 18 23:34:37.170737 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:37.170714    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbczd_05b8f250-4be6-4182-9706-303d1f6ee7b9/extractor/0.log"
Sep 18 23:34:38.517552 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:38.517525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-gmwfh"
Sep 18 23:34:40.399826 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:40.399779    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hrsdw_f9a1ca22-90f4-4d43-8bd5-f470269823ec/openshift-lws-operator/0.log"
Sep 18 23:34:41.182618 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:41.182594    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-575575dfd7-nrvzj_6b0d1682-4a41-4f3f-a6dc-2fe26d17aff6/manager/0.log"
Sep 18 23:34:41.524239 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:41.524207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"
Sep 18 23:34:41.596946 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:41.596914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7774f8b66-9w5pn_c4766513-3605-433b-b3f1-2c7b0e0cd98a/manager/0.log"
Sep 18 23:34:41.623157 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:41.623137    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-x7szn_f85225eb-ede0-4de8-841c-4a0f31721cf5/server/0.log"
Sep 18 23:34:42.682405 ip-10-0-141-13 kubenswrapper[2563]: I0918 23:34:42.682348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zhst7_a1ae4f53-da2d-4e99-ac77-cb536497b251/discovery/0.log"